Support dynamic configuration

[jolestar]

Please support reload configuration files and do not need restart server.
I think there are two approach:

1. Catch SIGHUP signal
2. Offer a http api

[dennyzhang]

Hi fluentbit, any idea or thinking about this request?

It would be great, if we can have this feature implemented.

[jasonkeene]

SIGHUP currently takes down the process:

Fluent-Bit v0.13.4
Copyright (C) Treasure Data

[engine] caught signal (SIGHUP)

It would be great if this reloaded the config without taking down the process.

[nikolay]

It would be nice to watch the configuration files as well and support hot configuration reloading as well.

[vinkdong]

If there is an API like prometheus /-/reload , it’s even better.

[discordianfish]

Or just use inotify to watch the confiug file and automatically reload. It's using inotify for the log anyway, right?

[jasonkeene]

Not that I would do this, but I could imagine users modifying their config file on the server and writing out progress. If you reload on writes like that it may load an invalid or partial config. Also, this breaks from most server software that receive an explicit signal like SIGHUP.

[edsiper]

What's the current problem of restarting Fluent Bit using systemd ?, I would like to learn the real need of config reload without restarting, as far as I can tell services that offer this always start a new process (behind the scenes).

The real complexity will be to add this handling on every single input plugin. Please elaborate the use-case.

[discordianfish]

@edsiper I'm running fluent-bit in a container on kubernetes, so it's not using systemd. Running another process manager in the container, at least on kubernetes, is an antipattern since you loose the process management features provided by kubernetes (restart/policies, monitoring, event log etc).

That being said, there is a case to be made for it being kubernetes's job to restart the fluent-bit container when the config changes. Something that's being discussed in kubernetes/kubernetes#22368

Currently the best option would be reloading on:

• inotify (people would need to write the config update atomically, but this is an established pattern and can be easily achieved by mv the file to the final location after creating it somewhere else)
• @vinkdong's suggestion of an http endpoint which can be used with https://github.com/jimmidyson/configmap-reload as a sidecar container (it would watch the config file via inotify and send the http request to reload)

Reloading on SIGHUP should also work but requires an alpha feature: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/

[nikolay]

@discordianfish The @edsiper's answer speaks volumes! I had so many problems with Fluent Bit that I switched to Filebeat, which is mature, written in Go so more people can contribute, and it has most of the features Fluent Bit is not even planning to have any time soon, including configuration reloading.

[donbowman]

I think adding an inotify on the config and restart would be good.

the next step would be to emulate grafana and have multiple config maps, one for each pod that wants to have custom log message formats. This means that fluent-bit watches for new configmaps to be written, and reads them as they happen. (see https://github.com/coreos/prometheus-operator).

e.g. in my pod i could have a 'fluent-parser' and have a single parser entry, which would be dynamically loaded.

the restart can be ok as long as the trackingoffsets is used. Otherwise its disasterous.

[nikolay]

@donbowman Filebeat already has multiple formats (let's say, Nginx access and error logs), and reloading works, too. Also, it has a super easy way to create your own parsing pipelines without any programming. And, most importantly, it has a very active and passionate team that works on it full-time. I'm sorry to be promoting Filebeat, but I invested almost one year in Fluent Bit and the return was only pain! It's not common knowledge, but Filebeat has very good built-in Kubernetes hints as well. The only built-in feature that is missing right now is a Prometheus metrics endpoint, but there's an open-source project, which I compiled into a 2MB-image sidecar, which exports Filebeat's rich JSON metrics to the Prometheus metrics format. And soon Filebeat will have native Prometheus metrics endpoint. Also, all the standard parsers feed namespaced metrics to Elasticsearch, so, you don't get messy metrics under the same name.

Hopefully, the Fluent Bit team reads this, looks into Filebeat, and starts putting more passion into delivering something that works with basic uses cases if not with more complex stuff.

If I was Treasure Data, I'd rewrite this in Go if not in Rust and make it work perfectly with Kubernetes and ignore all other stuff including my own niche platform. And then offer enterprise support plants. For example, why put configuration in files at the end of 2018, when it can read it directly from ConfigMaps, and watch them for updates?!

There's still a chance for this project as Elastic is door a tremendously poor job at advertising Filebeat and I don't think they still pay enough attention to Kubernetes.

[donbowman]

go, rust, c are the same level of abstraction. changing a language changes nothing from a user standpoint, just creates work without creating value. i wasnt comparing to other tools, just adding comments about best practise. kubernetes tools typically use files via config map, and are decoupled so they can start in any order at any time. the coreos Prometheus operator with grafana is an example

…

On Mon., Sep. 24, 2018, 23:19 Nikolay Kolev, ***@***.***> wrote: @donbowman <https://github.com/donbowman> Filebeat already has multiple formats (let's say, Nginx access and error logs), and reloading works, too. Also, it has a super easy way to create your own parsing pipelines without any programming. And, most importantly, it has a very active and passionate team that works on it full-time. I'm sorry to be promoting Filebeat, but I invested almost one year in Fluent Bit and the return was only pain! It's not common knowledge, but Filebeat has a very good built-in Kubernetes hints as well. The only built-in feature that it is missing right now is a Prometheus metrics endpoint, but there's an open-source project, which I compiled into a 2MB-image sidecar, which exports Filebeat's rich JSON metrics to the Prometheus metrics format. And soon Filebeat will have native Prometheus metrics endpoint. Also, all the standard parsers feed namespaced metrics to Elasticsearch, so, you don't get messy metrics under the same name. Hopefully, the Fluent Bit team reads this, looks into Filebeat, and starts putting more passion into delivering something that works with basic uses cases if not with more complex stuff. I was Treasure Data, I'll rewrite this in Go if not in Rust and make it work perfectly with Kubernetes and ignore all other stuff including my own niche platform. And then offer enterprise support plants. For example, why put configuration in files at the end of 2018, when it can read it directly from ConfigMaps, and watch them for updates?! There's still chance for this project as Elastic is door a tremendously poor job at advertising Filebeat and I don't think they still pay enough attention to Kubernetes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#365 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE5Ok111coXLlUfk2E4tf9hAtDYZ5OiTks5ueaCkgaJpZM4PA2JC> .

[edsiper]

I am not against configuration reloading, but understanding the why and it urgency helps to assign a priority, as you can guess with different interfaces in input, filters and outputs is not a straightforward change. As said above, for Kubernetes use-case config-reloading is an anti-pattern (immutable).

Now about Fluent Bit v/s other projects, I consider that is really good to learn from others, that's the way to go. We still have a plenty of room for improvements.

About programming languages, consensus is something that will unlikely exists and having different preferences is just fine. Users decide their tooling, hopefully they have choices, Fluent Bit is one of them.

[nikolay]

@donbowman I don't think C, C++, Rust, and Go have much in common outside of being compiled languages.