Get HHVM logs into logstash
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	Nov 27 2014, 2:12 AM

Description

The HHVM logs event stream is relayed from the HHVM servers to fluorine.eqiad.wmnet but is currently not stored in logstash.

There are 2 reasonable ways to add these log events to logstash:

Add HHVM to the existing log2udp packet processing done in logstash. This would be a fairly trivial change as I'm fairly certain that udp2log is already relaying these events to the logstash listener but the events are not being tagged for storage in Elasticsearch.
Add puppet configuration to forward the rsyslog event stream to logstash either via an intermediate relay on fluorine or by direct packets from the HHVM app servers. I have already created rules in the MW-Vagrant role::elk logstash role that does this and the event stream seems to be imported nicely with few intermediate processing steps needed.

I'd eventually like to kill off the use of log2udp relaying entirely as a logstash event feed mechanism, so I think that looking into the rsyslog event forwarding would be the better option.

Details

	Subject	Repo	Branch	Lines /-
	logstash: Forward syslog events for apache2 hhvm	operations/puppet	production	68 -3

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	hashar	T71590 rsync errors to beta cluster, inconsistent state after scap
Resolved	hashar	T71601 Log files on labs instance fill up disk (/var is only 2GB) (tracking)
Invalid	None	T71976 HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk
Resolved	hashar	T75262 Beta-cluster web server fills up /var/log with Apache logs
Resolved	bd808	T76119 Get HHVM logs into logstash

Event Timeline

bd808 created this task.Nov 27 2014, 2:12 AM

bd808 claimed this task.

bd808 raised the priority of this task from to High.

bd808 updated the task description. (Show Details)

bd808 added projects: Wikimedia-Logstash, HHVM, MediaWiki-Core-Team.

bd808 changed Security from none to None.

bd808 subscribed.

I marked this as a high priority because this really should have been done as soon as @ori had the hhvm event forwarding setup and the rapidly growing hhvm server base needs easier error monitoring.

bd808 mentioned this in T74275: no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs.Nov 27 2014, 2:17 AM

Change 176693 had a related patch set uploaded (by BryanDavis):
logstash: Forward syslog events for apache2 hhvm

https://gerrit.wikimedia.org/r/176693

Patch-For-Review

bd808 moved this task from Backlog to In Dev/Progress on the MediaWiki-Core-Team board.Dec 1 2014, 10:06 PM

I have made two tasks as child of this:

T75262: Beta-cluster web server fills up /var/log with Apache logs
T71976: HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk

I guess both would be resolved once logs are sent to logstash instead of being written on local disk.

In T76119#799627, @hashar wrote:

I have made two tasks as child of this:

T75262: Beta-cluster web server fills up /var/log with Apache logs
T71976: HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk

I guess both would be resolved once logs are sent to logstash instead of being written on local disk.

@hashar my initial patches are replicating the log event stream rather than diverting it, so they won't keep things from being written to disk on the individual hhvm hosts. If it's needed we could make followup patches that either conditionally (or maybe everywhere if that's reasonable) disable the local syslog output.

JanZerebecki subscribed.Dec 3 2014, 7:19 PM

Change 176693 merged by Yuvipanda:
logstash: Forward syslog events for apache2 hhvm

https://gerrit.wikimedia.org/r/176693

bd808 moved this task from In Dev/Progress to Done on the MediaWiki-Core-Team board.Dec 4 2014, 12:00 AM

bd808 removed a project: Patch-For-Review.

HHVM log stream is being sent to logstash via rsylog forwarding in both beta and production. Prod dashboard at https://logstash.wikimedia.org/#/dashboard/elasticsearch/hhvm

In T76119#799936, @bd808 wrote:

In T76119#799627, @hashar wrote:

I have made two tasks as child of this:

T75262: Beta-cluster web server fills up /var/log with Apache logs
T71976: HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk

I guess both would be resolved once logs are sent to logstash instead of being written on local disk.

@hashar my initial patches are replicating the log event stream rather than diverting it, so they won't keep things from being written to disk on the individual hhvm hosts. If it's needed we could make followup patches that either conditionally (or maybe everywhere if that's reasonable) disable the local syslog output.

Understood. That is why I made the tasks to remove the local logs to be blocked by this logstash one. Now that we have logs centralized, we can look at disabling local logs \O/ Thank you!

bd808 moved this task from Done to Archive on the MediaWiki-Core-Team board.Dec 8 2014, 10:55 PM

yuvipanda mentioned this in Unknown Object (Diffusion Commit).Dec 10 2014, 8:37 PM

yuvipanda mentioned this in rOPUPabc6646d0a81: logstash: Forward syslog events for apache2 hhvm.Dec 30 2014, 7:59 PM

bd808 moved this task from Backlog to Archive on the Wikimedia-Logstash board.Apr 22 2015, 3:25 AM

fgiunchedi added a project: observability.Aug 19 2019, 2:29 PM