Build a slim container for fluentd
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	Joe
	Sep 11 2017, 9:55 AM

Description

We need to create a fluentd container for our standardized pod setup.

It should be enough to just port https://github.com/fluent/fluentd-docker-image/blob/master/v0.14/debian/Dockerfile to use our stretch image.

I'm not a big fan of how fluentd gets installed here (via ruby gems!) and we might want to improve upon that.

Details

	Subject	Repo	Branch	Lines /-
	Add fluent-bit image	operations/docker-images/production-images	master	35 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T170453 FY2017/18 Program 6: Streamlined Service delivery
Resolved	akosiaris	T170108 Operations Q1 goal: Streamlined Service Delivery
Resolved	None	T170120 Standardize on the "default" pod setup
Resolved	None	T175527 Build a slim container for fluentd
Resolved	None	T175539 Build containers for statsd, prometheus-statsd-exporter

Event Timeline

Joe created this task.Sep 11 2017, 9:55 AM

Joe created subtask T175539: Build containers for statsd, prometheus-statsd-exporter.Sep 11 2017, 10:47 AM

we might want to improve upon that.

Yeah, definitely. Also, I don't have a concrete use case in mind, but I'm going to guess that there is also a chance that we'll need to create custom input plugins (at some point).

After some reasoning, I decided to go the following way:

create a fluentbit container for the standardized pod. This will mostly just blindly listen to messages incoming on a specific port, we will need to include a few parser configurations there, but the weight of the container and the footprint of the agent is way smaller. For an example dockerfile, see https://github.com/fluent/fluent-bit-docker-image/blob/master/0.12/Dockerfile
Have a daemonset running fluentd that catches all kube-related logs and whatever containers do not forward to fluentd. Possibly forward messages from fluentbit here *if* they need further/better processing, or directly to Elasticsearch. An example is here https://github.com/fluent/fluentd-kubernetes-daemonset/tree/master/docker-image/v0.12/debian-elasticsearch

Mentioned in SAL (#wikimedia-operations) [2017-09-15T15:21:29Z] <_joe_> uploaded td-agent-bit (fluentd-bit) to wikimedia-stretch T175527

Change 378260 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/docker-images/production-images@master] Add fluent-bit image

https://gerrit.wikimedia.org/r/378260

gerritbot added a project: Patch-For-Review.Sep 15 2017, 4:16 PM

Change 378260 merged by Giuseppe Lavagetto:
[operations/docker-images/production-images@master] Add fluent-bit image

https://gerrit.wikimedia.org/r/378260