I'm not exactly sure what is going here, but deployment-logstash03 is failing Puppet:
root@deployment-logstash03:/etc/logstash# run-puppet-agent Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Info: Caching catalog for deployment-logstash03.deployment-prep.eqiad.wmflabs Info: Unable to serialize catalog to json, retrying with pson Info: Applying configuration version '(a45e216e34) root - [LOCAL HACK] Allow external access from anywhere to parsoid port 80 for CI purposes' Notice: The LDAP client stack for this host is: classic/sudoldap Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: classic/sudoldap' Error: Execution of '/usr/local/bin/logstash-config-test /etc/logstash/conf.d/90-output-kafka-deprecated.conf20210305-12589-uwgdko' returned 1: 2021-03-05 08:00:53,919 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root" WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console Error: /Stage[main]/Profile::Logstash::Collector/Logstash::Output::Kafka[deprecated]/Logstash::Conf[output-kafka-deprecated]/File[/etc/logstash/conf.d/90-output-kafka-deprecated.conf]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/local/bin/logstash-config-test /etc/logstash/conf.d/90-output-kafka-deprecated.conf20210305-12589-uwgdko' returned 1: 2021-03-05 08:00:53,919 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root" WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console (corrective) Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/90-output-kafka-deprecated.conf] has failures: true Warning: /Stage[main]/Logstash/Service[logstash]: Skipping because of failed dependencies Info: Stage[main]: Unscheduling all events on Stage[main] Notice: Applied catalog in 16.72 seconds
logstash-beta stopped receiving errors when debugging, T241481: deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd