Page MenuHomePhabricator

deployment-logstash03 puppet errors
Closed, DeclinedPublic

Description

I'm not exactly sure what is going here, but deployment-logstash03 is failing Puppet:

root@deployment-logstash03:/etc/logstash# run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for deployment-logstash03.deployment-prep.eqiad.wmflabs
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(a45e216e34) root - [LOCAL HACK] Allow external access from anywhere to parsoid port 80 for CI purposes'
Notice: The LDAP client stack for this host is: classic/sudoldap
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: classic/sudoldap'
Error: Execution of '/usr/local/bin/logstash-config-test /etc/logstash/conf.d/90-output-kafka-deprecated.conf20210305-12589-uwgdko' returned 1: 2021-03-05 08:00:53,919 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Error: /Stage[main]/Profile::Logstash::Collector/Logstash::Output::Kafka[deprecated]/Logstash::Conf[output-kafka-deprecated]/File[/etc/logstash/conf.d/90-output-kafka-deprecated.conf]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/local/bin/logstash-config-test /etc/logstash/conf.d/90-output-kafka-deprecated.conf20210305-12589-uwgdko' returned 1: 2021-03-05 08:00:53,919 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console (corrective)
Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/90-output-kafka-deprecated.conf] has failures: true
Warning: /Stage[main]/Logstash/Service[logstash]: Skipping because of failed dependencies
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 16.72 seconds

logstash-beta stopped receiving errors when debugging, T241481: deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd

Event Timeline

Mentioned in SAL (#wikimedia-releng) [2021-03-05T11:48:27Z] <Majavah> live hack beta puppetmaster to fix hopefully trust store location; T276521 and possibly others

modules/profile/manifests/logstash/collector.pp had a path to Kafka trust store that didn't exist. I applied a beta-only hack to change it to see if it works. The machine now runs puppet properly.

This machine will be decommissioned shortly (T283013)