Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Predefined server-snippet that can only be reference and used in ingress #11259

Open
Anddd7 opened this issue Apr 14, 2024 · 7 comments
Open
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@Anddd7
Copy link
Contributor

Anddd7 commented Apr 14, 2024

What do you want to happen?

We want to host a set of multi-tenant applications in our cluster. There are a lot of similar server-snippets in those ingresses, e.g. filter header, redirection. Because everyone shares a same nginx-controller, so we want to limit or even stop the use of snippets to secure our platform.

So we're wondering if it's possible to have:

  • AdminOps creates pre-defined snippets in a custom nginx.tmpl
  • AdminOps rollouts those 'features' by replacing the nginx.tmpl
  • DevOps use custom annotations to call those snippets

So, we can ensure server-snippets is used under restricted conditions.

Is there currently another issue associated with this?

No

Does it require a particular kubernetes version?

No (only nginx template rendering)

If this is actually about documentation, uncomment the following block

No

@Anddd7 Anddd7 added the kind/feature Categorizes issue or PR as related to a new feature. label Apr 14, 2024
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Apr 14, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@longwuyuan
Copy link
Contributor

There is no interest in the project to keep adding new annotations on-demand. The reason is that the volume of unique important use-cases is too large to cater to all expectations. Meaning if annotations are created for all requirements, then it becomes a combination of limited use of huge maintenance effort, of non-generic annotations or other features.

Secondly, the planning and execution of the Gateway API is already underway. The Gateway API will impact many expectations and alter designs. So adding annotations is not a improvement.

Thank you for your other contribution of gRPC_timeout annotation. That timeout annotation exists for HTTP but not for gRPC. So its a generic use-case for all gRPC deployments and hence its a improvement.

Wait for comments from others.

@Anddd7
Copy link
Contributor Author

Anddd7 commented Apr 14, 2024

sure, i don't want to add annotations directly in this repo, that will bring chaos (like you said). 😄

i just want a mechanism that support custom annotations at runtime, like https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/custom-annotations/.

  • the nginx.tmpl can be mounted to controller
  • the annotations can be added to ingress, and only effect when it has been defined in nginx.tmpl

so, we don't need to create a full set of codes for annotation, just expose a map of variables for tmpl.

(let's see others 👍 )

@longwuyuan
Copy link
Contributor

I don't think this project's controller has the design & architecture for supporting anything resembling dynamic runtime annotations.

Wait for other comments.

Copy link

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

@github-actions github-actions bot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label May 15, 2024
@tylermichael
Copy link

I have a similar use-case but for configuration-snippet. We have one ingress where we want to alter headers before the request is proxied to the backend. This requires us to enable allowSnippetAnnotations in our helm chart, but we are concerned about that because of the corresponding CVE.

I think having a feature such as this is a reasonable compromise. It would let you pre-define trusted custom snippets, and then opt-in usage of them where required. This contrasts how configuration-snippet and server-snippet work because they provide more surface area for arbitrary code to be set.

This could look like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: "false"
  enable-opentelemetry: "true"
  configuration-snippets: |
    - name: "add-foo-header"
      snippet: |
          add_header Foo "Bar";
    - name: "do-not-proxy-cookies"
      snippet: |
          proxy_set_header Cookie "";
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-configuration-snippets: "add-foo-header,do-not-proxy-cookies"
  name: foo
  namespace: default
spec:
  ingressClassName: ingress-nginx
  rules:
  - host: foo.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: backend
            port: 
              number: 80

@tylermichael
Copy link

I believe this is partially possible today, it requires you to mount a custom nginx template file, then you can add code like they have here in the NGINX ingress documentation.

Instead of doing {{if index $.Ingress.Annotations "custom.nginx.org/feature-a"}}, you would do something like (simplified) around here:

{{ $enabledConfigSnippets := index $ing.Annotations "nginx.ingress.kubernetes.io/enable-configuration-snippets" }}

{{ if not (empty $enabledConfigSnippets ) }}

 # use `split`[0], and look for exact values
{{ if contains $enabledConfigSnippets "do-not-proxy-cookies" }}
proxy_set_header Cookie "";
{{ end }}

{{ end }}

I think this could be simplified if it were a first class feature, plus you wouldn't need to maintain your own nginx template.

Links:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
Development

No branches or pull requests

4 participants