Fluentd input plugin to pull log from rest api.
Many of modern server application offer status reporting API via http (even 'fluentd' too). This plugin will help to gathering status log from these status api.
$ gem install fluent-plugin-http-pull
Add following line to your Gemfile:
gem "fluent-plugin-http-pull"And then execute:
$ bundle
In your Fluentd configuration, use @type http_pull.
tag, url, interval, format is mandatory configuration.
<source>
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
format json
</source>
# 2017-05-17 21:41:47.872951000 0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
In this example, a response of your infrastructure parsed as json.
A result contains url, status, message. message is a response of your infrastructure.
{
"url": "http://your-infrastructure/api/status.json",
"status": 200,
"message": {
// response of your infra structure
}
}
You can found more examples in this document.
If you need only http status code, not response body, You can turn off response
body parser to set status_only is true. Remember that format is mandatory.
In this case, you must set format is none.
<source>
@type http_pull
tag fluentd.status
url http://your-infrastructure/healthcheck
interval 1s
status_only true
format none
</source>
# 2017-05-17 21:41:47.872951000 0900 status: {"url":"http://yourinfrastructure/healthcheck","status":200}
You can set the User Agent by specifying the agent option.
<source>
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
format json
agent infrastructure_monitor
</source>
# 2017-05-17 21:41:47.872951000 0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
If your infrastructure is protected by HTTP basic auth,
You can use user, password options to provide authentication information.
<source>
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
format json
user foo
password bar
</source>
# 2017-05-17 21:41:47.872951000 0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
You can send your requests via proxy server.
<source>
@type http_pull
tag status
url http://your-infrastructure/api/status.json
proxy http://your-proxy-server:3128
interval 1s
format json
</source>
# 2017-05-17 21:41:47.872951000 0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
If you wish to monitoring not only response body but also response header,
provide name of header in response_header sections.
<source>
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
format json
<response_header>
header Content-Type
</response_header>
<response_header>
header Content-Length
</response_header>
</source>
# 2017-05-17 21:41:47.872951000 0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
The custom request header also supported. This will help you when your infrastructure needs authentication headers or something like that.
<source>
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
format json
<request_header>
header API_ACCESS_KEY
value hatsune
</request_header>
<response_header>
header API_ACCESS_KEY_SECRET
value miku
</response_header>
</source>
# 2017-05-17 21:41:47.872951000 0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
If your infrastructure has https endpoints, just use https url for monitoring them.
If your infrastructure has https endpoints secured by self signed certification,
you can provide custom certification file via ca_path, ca_file option.
<source>
@type http_pull
tag status
url https://your-infrastructure/api/status.json
interval 1s
ca_path /somewhere/ca/stored
ca_file /somewhere/ca/stored/server.crt
format json
</source>
# 2017-05-17 21:41:47.872951000 0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
And, disable SSL verification also supported. (not recommended)
<source>
@type http_pull
tag status
url https://your-infrastructure/api/status.json
interval 1s
verify_ssl false
format json
</source>
# 2017-05-17 21:41:47.872951000 0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
The tag of the event.
The url of remote server.
The user agent string of request.
The interval time between periodic request.
The format of the response body. Due to limitation of current implement it is
always required regardless status_only option.
http_pull uses parse plugin to parse the response body. See
parser article
for more detail.
If status_only is true, body is not parsed.
The http request method for each requests. Avaliable options are listed below.
getpostdelete
If status_only is true, http_method was override to head
The timeout of each request.
The HTTP proxy URL to use for each requests
The user for basic auth
The password for basic auth
The name of response header for capture.
The name, value pair of custom reuqest header.
When false, SSL verification is ignored.
The absolute path of directory where ca_file stored. Should be used with ca_file.
The absolute path of ca_file. Should be used with ca_path.
This plugin is tested with the following version combination of ruby and fluentd.
| http_pull | ruby | fluentd |
|---|---|---|
| <= v0.7.0 | 2.3 | v0.14.x |
| v0.8.0 | 2.3 | >= v1.0.0 |
| v0.8.1 | 2.3 | <= v1.14.4 |
| v0.8.2 | 2.3 | <= v1.14.4 |
| v0.8.3 | 2.3 | <= v1.14.4 |
- Copyright(c) 2017- filepang
- License
- Apache License, Version 2.0