[bitnami/fluentd] Unreadable log mounts #1905
Comments
|
I have another Fluentd daemonset running as well - could they be interfering with one another? |
|
Hi, Did you deploy it with the security group as root? |
|
Hey @javsalgar, yeah here's my current config: forwarder:
## Enable forwarder daemonset
##
enabled: true
## K8s Security Context for forwarder pods
## https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
enabled: true
runAsUser: 0
fsGroup: 0 |
|
Looks like it's being mounted as |
|
I've also verified that |
|
I have the same problem. Error log in forwarderForwarder container current userPart configuration of daemonset fluentPart of the configuration of forwarder in values.yamlLog path relationship |
The question would be, is that path also mounted in the container? If not, the symlink would not work |
|
By default, |
|
Could you also show me the permissions of the symlinked file inside the container? Are you able to read it with cat for example? I'm trying to understand the reason of not being readable |
|
This is the log of what I did inside the container: |
|
Same here. Also with AWS EKS. fluentd runs with a After an user switch i was not able to |
|
Hi, You showed me the permissions of the symlink, but not the permissions of the symlinked file. It definitely seems that the permissions of these logs are restrictive so the fluentd user is unable to read form them. One option that comes to mind is to change the permissions in the host system so the logs are readable. I am not sure about forcing the fluentd process to be run as pure root. |
|
|
Did the PR fix the issue? |
|
Unfortunately no :-( Same behaviour with: |
|
Somehow Promtail works just fine. They are doing similiar things..https://github.com/grafana/loki/blob/master/production/helm/promtail/values.yaml#L91-L105 |
|
I don't get it....we can't be the first that are running fluentd on AWS EKS 😂 |
|
I think it's directly related to the |
|
I just saw that Promtail runs as root as well. I will try using |
|
In this kind of scenarios what we could do is to allow initContainers that would change the permissions of your mounted files or, instead, change the permissions in your host system. We try to avoid using |
|
Understandably so @javsalgar -- I'd avoid changing permissions on the host system as much as possible as well, especially in a managed k8s environment like EKS, but interested to see how the |
|
The thing is that the root container would also change permissions on the host system because essentially, it would just chmod the mounted folders. I'm not 100% sure what the best way would be in order to avoid a |
|
Setting FLUENT_UID to 0 didn't work for me. I had to add the following to forwarder.extraEnv Since only the root user has access to /var/lib/docker/containers on the host. |
|
Thanks for sharing! I believe it would be interesting adding this to the chart documentation. |
|
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback. |
|
Not stale |
javsalgar
added
the
on-hold
|
See #2253. |
|
Just to give my point of view, executing the main process of the application as root goes against our security conventions. It is true that in certain environments, if you don't do that, fluentd is totally unusable. But it is decision of the administrator how to workaround that issue. I wouldn't add the workaround suggested by @brettcat directly in the I'd rather add it to the chart README.md. Thoughts @javsalgar? |
I don't see the match with certain environments at all.
as reported in the Prerequisites:
Having the above defaults implies that files mounted in volumes are only readable by root regardless from the environment you choose, so either you add the variables to be compliant with chart defaults or the chart as is will be unusable. |
|
Hi, As far as I know, I think that the volumes could be readable by other user different from root. However I'm not entirely sure if fsGroup would apply correctly to hostPath volumes. Anyways, in the chart there is an |
|
Thanks @javsalgar - I think we have the several we need to make that change: @r3kzi - did these env variables solve your use case as well? |
|
Exactly, this is not user related either. As reported here The FLUENTD_DAEMON_USER / FLUENTD_DAEMON_GROUP variables are the only way to make this chart usable at the moment. |
Well, the documentation only says writable. Theoretically you could make the logs folder readable by other users and it should work. But I will open a task for evaluating and implementing the option for the |
The documentation says writable, but as a matter of fact it's also readable only by root because:
I don't think this would add any extra benefit to the chart, FLUENTD_DAEMON_USER / FLUENTD_DAEMON_GROUP variables already make the chart work as workaround. |
|
Hi. I created this PR for discussion #2323 |
|
The issue was closed as the PR was merged. lease do not hesitate to re-open it if you find further issues. |
|
Hey |
|
It turns out that an extra hostPath |
|
Hi, Thanks for letting us know! :) It will be helpful for other OKE users. |
carrodher
removed
the
on-hold
It works, thanks.. |
Which chart:
Fluentd 0.4.10
Description
Mounted logs are unreadable. It seems similar to this issue: https://stackoverflow.com/questions/51671212/fluentd-log-unreadable-it-is-excluded-and-would-be-examined-next-time
I have already set the FLUENT_UID to '0'. What other information do you need to find out what else I could be doing wrong?
Steps to reproduce the issue:
Describe the results you received:
Describe the results you expected:
Additional information you deem important (e.g. issue happens only occasionally):
Version of Helm and Kubernetes:
helm version:kubectl version:The text was updated successfully, but these errors were encountered: