[dev.icinga.com #10655] API queries cause memory leaks #3677

icinga-migration · 2015-11-17T01:30:28Z

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10655

Created by vvv on 2015-11-17 01:30:28 00:00

Assignee: mfriedrich
Status: Resolved (closed on 2016-02-03 12:56:49 00:00)
Target Version: 2.4.2
Last Update: 2016-02-23 09:58:18 00:00 (in Redmine)

Icinga Version: 2.4.0
Backport?: Already backported
Include in Changelog: 1

API Query opens threads but the thread does not die after the api query is done.

Checking for icinga2 threads:

icinga@icinga-np:/home/vvv$ for i in {1..100000}; do ps -eLf | grep "no-stack" | wc -l; sleep 5; done
17
17
17
17
17
17
17
22
32
42
52
62
72
82
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
84
85
85
85
85
85
85
85
86
86
86
86
86
86
86
86
86
86
87
87
87
87
87
87
91
94
97
100
103
107
110
113
116
119
123

Query used for API

VVV-MacBook-Pro:~ vvv$ for i in {1..100000};do curl -k https://root:icinga@icinga-np:5665/v1/objects/hosts?host=icinga-np; done

After killing the continuous curl call the thread count went steady at 123.

Changesets

2016-01-13 09:30:38 00:00 by (unknown) 7f4af89

Ensure that HttpServerConnections are properly closed

fixes #10655

2016-02-03 08:54:31 00:00 by (unknown) 2dc385e

Fix memory/thread leak in the HttpServerConnection class

fixes #10655

2016-02-03 11:58:49 00:00 by (unknown) cfe9849

Fix memory leak in Stream::Close

refs #10655

2016-02-03 12:02:49 00:00 by (unknown) caf3380

Fix memory leak in Stream::Close

refs #10655

2016-02-23 08:38:05 00:00 by (unknown) b6a799d

Ensure that HttpServerConnections are properly closed

fixes #10655

2016-02-23 08:38:23 00:00 by (unknown) 9a0107d

Fix memory/thread leak in the HttpServerConnection class

fixes #10655

2016-02-23 08:38:24 00:00 by (unknown) 8af6ad7

Fix memory leak in Stream::Close

refs #10655

Relations:

relates #10655
relates #10655
duplicates #10655
duplicates #10655

The text was updated successfully, but these errors were encountered:

icinga-migration · 2015-11-25T14:25:25Z

Updated by mfriedrich on 2015-11-25 14:25:25 00:00

Category set to API

icinga-migration · 2015-11-28T08:13:56Z

Updated by jonan on 2015-11-28 08:13:56 00:00

I can also confirm resource exhaustion using the API when running the following Icinga environment:

Icinga version: v2.4.1-42-g6a83703 on Ubuntu 14.04

Hitting any api call seems to do it. You can reproduce as follows:

Monitor the Icinga thread count with:

    $ watch -n1 'ps -o nlwp '

Query API using Apache bench:

    $ ab -A root:icinga -n 17000 -c 100 https://localhost:5665/v1

Icinga dies after hitting the max processes limit of 16384.

Logs show the following errors when failure begins:

> warning/HttpServerConnection: Error while reading Http request: Error: boost::thread_resource_error: Resource temporarily unavailable

> critical/checker: Exception occured while checking 'host-22!ssh': Error: Function call 'fork' failed with error code 11, 'Resource temporarily unavailable'

icinga-migration · 2015-11-30T16:59:20Z

Updated by mfriedrich on 2015-11-30 16:59:20 00:00

Status changed from New to Assigned
Assigned to set to mfriedrich
Target Version set to Backlog

I'm able to reproduce the issue, though I need to look into it in deep when other projects are finished.

mbmif /usr/local/icinga2 (master) # lldb -- /usr/local/icinga2/lib/icinga2/sbin/icinga2 daemon
(lldb) target create "/usr/local/icinga2/lib/icinga2/sbin/icinga2"
Current executable set to '/usr/local/icinga2/lib/icinga2/sbin/icinga2' (x86_64).
(lldb) settings set -- target.run-args  "daemon"
(lldb) r
Process 68017 launched: '/usr/local/icinga2/lib/icinga2/sbin/icinga2' (x86_64)
[2015-11-30 12:32:50  0100] information/cli: Icinga application loader (version: v2.4.1-47-gab9218e)
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/icinga2.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/constants.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/zones.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/itl
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command.conf
[2015-11-30 12:32:50  0100] information/Utility: Loading library 'libmethods.dylib'
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command-icinga.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/timeperiod.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command-plugins.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/databases.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/hardware.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/ipmi.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/network-components.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/operating-system.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/virtualization.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/vmware.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/web.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/api.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/checker.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/command.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/compatlog.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/graphite.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/ido-mysql.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/livestatus.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/mainlog.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/notification.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/perfdata.conf
[2015-11-30 12:32:50  0100] information/Utility: Loading library 'libchecker.dylib'
[2015-11-30 12:32:50  0100] information/Utility: Loading library 'libcompat.dylib'
[2015-11-30 12:32:50  0100] information/Utility: Loading library 'libcompat.dylib'
[2015-11-30 12:32:50  0100] information/Utility: Loading library 'libperfdata.dylib'
[2015-11-30 12:32:50  0100] information/Utility: Loading library 'libdb_ido_mysql.dylib'
[2015-11-30 12:32:50  0100] information/Utility: Loading library 'liblivestatus.dylib'
[2015-11-30 12:32:50  0100] information/Utility: Loading library 'libnotification.dylib'
[2015-11-30 12:32:50  0100] information/Utility: Loading library 'libperfdata.dylib'
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/api-users.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/app.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/aws.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/commands.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/comments.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/downtimes.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/groups.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/hosts.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/notifications.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/satellite.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/services.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/templates.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/test.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/timeperiods.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/users.conf
[2015-11-30 12:32:50  0100] information/config: []
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/10237.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/8183.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/dep.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/groups.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/mw.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/out.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/random.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/include.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/include.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/../active.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/comments/mbmif.int.netways.de!bla.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/mbmif.int.netways.de!load!mbmif.int.netways.de-1448872589-2.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/bumsti1.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com1.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com2.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com3.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com4.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/i-42866686.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/i-43866687.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/michiathome1.conf
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/services/bumsti1!ping7.conf
[2015-11-30 12:32:50  0100] information/ConfigItem: Committing config items
[2015-11-30 12:32:50  0100] information/ApiListener: My API identity: mbmif.int.netways.de
[2015-11-30 12:32:50  0100] warning/ConfigObject: Ignoring config object 'mbmif.int.netways.de-1448445272-0' of type 'Downtime' due to errors: Error: Validation failed for object 'master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0' of type 'Downtime'; Attribute 'host_name': Object 'master.dm.netways.de' of type 'Host' does not exist.
Location: in /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf: 1:0-1:66
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf(1): object Downtime "mbmif.int.netways.de-1448445272-0" ignore_on_error {
                                                                                                                                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf(2):  author = "icingaadmin"
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf(3):  comment = "Scheduled downtime for backup"

[2015-11-30 12:32:50  0100] warning/ConfigObject: Ignoring config object 'mbmif.int.netways.de-1448439527-0' of type 'Downtime' due to errors: Error: Validation failed for object 'master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0' of type 'Downtime'; Attribute 'host_name': Object 'master.dm.netways.de' of type 'Host' does not exist.
Location: in /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf: 1:0-1:66
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf(1): object Downtime "mbmif.int.netways.de-1448439527-0" ignore_on_error {
                                                                                                                                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf(2):  author = "icingaadmin"
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf(3):  comment = "Scheduled downtime for backup"

[2015-11-30 12:32:50  0100] warning/ConfigObject: Ignoring config object 'foo' of type 'Comment' due to errors: Error: Comment 'mbmif.int.netways.de!abc!foo' references a host/service which doesn't exist.
Location: in /usr/local/icinga2/etc/icinga2/conf.d/comments.conf: 1:0-1:35
/usr/local/icinga2/etc/icinga2/conf.d/comments.conf(1): object Comment "foo" ignore_on_error {
                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/icinga2/etc/icinga2/conf.d/comments.conf(2):   host_name = NodeName
/usr/local/icinga2/etc/icinga2/conf.d/comments.conf(3):   service_name = "abc"

[2015-11-30 12:32:50  0100] warning/ApplyRule: Apply rule 'satellite-host' (in /usr/local/icinga2/etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 Endpoint.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 Zone.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 ApiListener.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 FileLogger.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 ApiUser.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 70 Dependencies.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 Downtime.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 44 Services.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 3 ServiceGroups.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 3 TimePeriods.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 14 Users.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 3 UserGroups.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 26 Hosts.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 6 HostGroups.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 11 Notifications.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 154 CheckCommands.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 Comment.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 CompatLogger.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 PerfdataWriter.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 LivestatusListener.
[2015-11-30 12:32:50  0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2015-11-30 12:32:50  0100] information/ScriptGlobal: Dumping variables to file '/usr/local/icinga2/var/cache/icinga2/icinga2.vars'
[2015-11-30 12:32:50  0100] information/ConfigObject: Restoring program state from file '/usr/local/icinga2/var/lib/icinga2/icinga2.state'
[2015-11-30 12:32:50  0100] information/ConfigObject: Restored 353 objects. Loaded 0 new objects without state.
[2015-11-30 12:32:50  0100] information/ConfigItem: Triggering Start signal for config items
[2015-11-30 12:32:50  0100] information/ApiListener: Adding new listener on port '5665'
[2015-11-30 12:32:50  0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2015-11-30 12:32:50  0100] information/LivestatusListener: Created UNIX socket in '/usr/local/icinga2/var/run/icinga2/cmd/livestatus'.
[2015-11-30 12:32:50  0100] information/ConfigItem: Activated all objects.
[2015-11-30 12:32:50  0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/modified-attributes.conf
[2015-11-30 12:32:50  0100] information/ApiListener: New client connection (no client certificate)
[2015-11-30 12:32:50  0100] information/HttpServerConnection: Request: GET /v1/objects/hosts?host[]=mbmif.int.netways.de (root)
[2015-11-30 12:32:50  0100] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.0')
[2015-11-30 12:32:50  0100] information/ApiListener: New client connection (no client certificate)
[2015-11-30 12:32:50  0100] information/HttpServerConnection: Request: GET /v1/objects/hosts?host[]=mbmif.int.netways.de (root)

...

[2015-11-30 12:34:22  0100] information/HttpServerConnection: Request: GET /v1/objects/hosts?host[]=mbmif.int.netways.de (root)
[2015-11-30 12:34:22  0100] information/ApiListener: New client connection (no client certificate)
[2015-11-30 12:34:22  0100] information/HttpServerConnection: Request: GET /v1/objects/hosts?host[]=mbmif.int.netways.de (root)
[2015-11-30 12:34:22  0100] information/ApiListener: New client connection (no client certificate)
[2015-11-30 12:34:22  0100] warning/HttpServerConnection: Error while reading Http request: Error: boost::thread_resource_error: Resource temporarily unavailable


    (0) 2   libbase.dylib                       0x000000010086391a _ZN5boost6thread12start_threadEv   122
    (1) 3   libbase.dylib                       0x0000000100860634 _ZN5boost12thread_group13create_threadINS_3_bi6bind_tIvNS_4_mfi3mf0IvN6icinga9WorkQueueEEENS2_5list1INS2_5valueIPS7_EEEEEEEEPNS_6threadET_   196
    (2) 4   libbase.dylib                       0x0000000100808cd2 _ZN6icinga9WorkQueue7EnqueueERKN5boost8functionIFvvEEEb   194
    (3) 5   libremote.dylib                     0x0000000100440c18 _ZN6icinga20HttpServerConnection14ProcessMessageEv   360
    (4) 6   libremote.dylib                     0x00000001004405b8 _ZN6icinga20HttpServerConnection20DataAvailableHandlerEv   56
    (5) 7   libremote.dylib                     0x00000001003f91ed _ZN6icinga20HttpServerConnection5StartEv   413
    (6) 8   libremote.dylib                     0x00000001003f6c9f _ZN6icinga11ApiListener24NewClientHandlerInternalERKN5boost13intrusive_ptrINS_6SocketEEERKNS_6StringENS_14ConnectionRoleE   1647
    (7) 9   libremote.dylib                     0x00000001003f54d9 _ZN6icinga11ApiListener16NewClientHandlerERKN5boost13intrusive_ptrINS_6SocketEEERKNS_6StringENS_14ConnectionRoleE   25
    (8) 10  libboost_thread-mt.dylib            0x0000000100043715 _ZN5boost12_GLOBAL__N_112thread_proxyEPv   53
    (9) 11  libsystem_pthread.dylib             0x00007fff86b599b1 _pthread_body   131
    (10) 12  libsystem_pthread.dylib             0x00007fff86b5992e _pthread_body   0
    (11) 13  libsystem_pthread.dylib             0x00007fff86b57385 thread_start   13



    (0) Handling new API client connection

Context:
    (0) Handling new API client connection

Process 68017 stopped
* thread #14: tid = 0x25ca81, 0x0000000100441f52 libremote.dylib`icinga::HttpRequest::Parse(this=0x0000000106915668, src=0x0000000106915ae0, may_wait=false)   898 at httprequest.cpp:142, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000100441f52 libremote.dylib`icinga::HttpRequest::Parse(this=0x0000000106915668, src=0x0000000106915ae0, may_wait=false)   898 at httprequest.cpp:142
   139                  return false;
   140              }
   141
-> 142              m_Body->Write(src.Buffer, length_indicator);
   143              src.DropData(length_indicator);
   144              Complete = true;
   145              return true;
(lldb)

  thread #2074: tid = 0x25faba, 0x00007fff8ac86f5e libsystem_kernel.dylib`__psynch_cvwait   10
* thread #2075: tid = 0x25fabd, 0x00007fff8ac86f5e libsystem_kernel.dylib`__psynch_cvwait   10
(lldb) thr sel 2074
* thread #2074: tid = 0x25faba, 0x00007fff8ac86f5e libsystem_kernel.dylib`__psynch_cvwait   10
    frame #0: 0x00007fff8ac86f5e libsystem_kernel.dylib`__psynch_cvwait   10
libsystem_kernel.dylib`__psynch_cvwait:
->  0x7fff8ac86f5e < 10>: jae    0x7fff8ac86f68            ; < 20>
    0x7fff8ac86f60 < 12>: movq   %rax, %rdi
    0x7fff8ac86f63 < 15>: jmp    0x7fff8ac823ef            ; cerror_nocancel
    0x7fff8ac86f68 < 20>: retq
(lldb) up
frame #1: 0x00007fff86b5a73d libsystem_pthread.dylib`_pthread_cond_wait   767
libsystem_pthread.dylib`_pthread_cond_wait:
    0x7fff86b5a73d < 767>: movl   �x, �x
    0x7fff86b5a73f < 769>: movq   %gs:0x0, %rdi
    0x7fff86b5a748 < 778>: movl   %r12d, %esi
    0x7fff86b5a74b < 781>: callq  0x7fff86b5a3d5            ; _pthread_testcancel
(lldb)
frame #2: 0x000000010085efeb libbase.dylib`boost::condition_variable::wait(this=0x000000010346cf58, m=0x00007000416f7ce8)   59 at condition_variable.hpp:73
   70               detail::interruption_checker check_for_interruption(&internal_mutex,&cond);
   71               guard.activate(m);
   72               do {
-> 73                 res = pthread_cond_wait(&cond,&internal_mutex);
   74               } while (res == EINTR);
   75   #else
   76               //boost::pthread::pthread_mutex_scoped_lock check_for_interruption(&internal_mutex);
(lldb)
frame #3: 0x0000000100857fe7 libbase.dylib`icinga::WorkQueue::WorkerThreadProc(this=0x000000010346cf08)   807 at workqueue.cpp:195
   192
   193      for (;;) {
   194          while (m_Tasks.empty() && !m_Stopped)
-> 195              m_CVEmpty.wait(lock);
   196
   197          if (m_Stopped)
   198              break;
(lldb) p *this
(icinga::WorkQueue) $4 = {
  m_ID = 2035
  m_ThreadCount = 1
  m_Spawned = true
  m_Mutex = {
    m = (__sig = 1297437786, __opaque = "")
  }
  m_CVEmpty = {
    internal_mutex = (__sig = 1297437786, __opaque = "")
    cond = (__sig = 1129270852, __opaque = "")
  }
  m_CVFull = {
    internal_mutex = (__sig = 1297437786, __opaque = "")
    cond = (__sig = 1129270852, __opaque = "")
  }
  m_CVStarved = {
    internal_mutex = (__sig = 1297437786, __opaque = "")
    cond = (__sig = 1129270852, __opaque = "")
  }
  m_Threads = {
    threads = size=1 {
      [0] = 0x0000000105e20390
    }
    m = {
      state = {
        shared_count = 0
        exclusive = false
        upgrade = false
        exclusive_waiting_blocked = false
      }
      state_change = {
        m = (__sig = 1297437786, __opaque = "")
      }
      shared_cond = {
        internal_mutex = (__sig = 1297437786, __opaque = "")
        cond = (__sig = 1129270852, __opaque = "")
      }
      exclusive_cond = {
        internal_mutex = (__sig = 1297437786, __opaque = "")
        cond = (__sig = 1129270852, __opaque = "")
      }
      upgrade_cond = {
        internal_mutex = (__sig = 1297437786, __opaque = "")
        cond = (__sig = 1129270852, __opaque = "")
      }
    }
  }
  m_MaxItems = 0
  m_Stopped = false
  m_Processing = 0
  m_Tasks = size=0 {}
  m_ExceptionCallback = {
    boost::function1 = {
      boost::function_base = {
        vtable = 0x0000000000000000
        functor = {
          obj_ptr = 0x0000000000000000
          type = {
            type = 0x0000000000000000
            const_qualified = false
            volatile_qualified = false
          }
          func_ptr = 0x0000000000000000
          bound_memfunc_ptr = (memfunc_ptr = 0x0000000000000000, obj_ptr = 0x0000000000000000)
          obj_ref = (obj_ptr = 0x0000000000000000, is_const_qualified = false, is_volatile_qualified = false)
          data = '\0'
        }
      }
    }
  }
  m_Exceptions = size=0 {}
  m_StatusTimer = {
    px = 0x0000000105b7abb0
  }
}

icinga-migration · 2015-12-15T11:10:54Z

Updated by mfriedrich on 2015-12-15 11:10:54 00:00

Relates set to 10839

icinga-migration · 2015-12-18T11:22:03Z

Updated by mfriedrich on 2015-12-18 11:22:03 00:00

Priority changed from Normal to High

icinga-migration · 2016-01-07T12:14:46Z

Updated by peikk0 on 2016-01-07 12:14:46 00:00

I have the same issue on FreeBSD 10.2 with version 2.4.1

icinga-migration · 2016-01-11T13:05:58Z

Updated by mfriedrich on 2016-01-11 13:05:58 00:00

Target Version changed from Backlog to 2.5.0

We should fix that, though I currently don't have an idea how to do it. Will discuss that for 2.5 at least.

icinga-migration · 2016-01-13T09:31:16Z

Updated by gbeutner on 2016-01-13 09:31:16 00:00

Assigned to changed from mfriedrich to gbeutner

icinga-migration · 2016-01-13T09:31:24Z

Updated by Anonymous on 2016-01-13 09:31:24 00:00

Status changed from Assigned to Resolved
Done % changed from 0 to 100

Applied in changeset 7f4af89.

icinga-migration · 2016-01-13T09:31:45Z

Updated by gbeutner on 2016-01-13 09:31:45 00:00

Please re-test this against the current Git master.

icinga-migration · 2016-01-13T09:55:09Z

Updated by mfriedrich on 2016-01-13 09:55:09 00:00

Status changed from Resolved to Assigned

The thread count stays the same after stopping the requests. Needs further investigation.

$ for i in {1..100000};do curl -k https://root:icinga@localhost:5665/v1/objects/hosts; done

mbmif ~ # ps M `pidof icinga2` | wc -l | xargs
270

icinga-migration · 2016-01-22T15:58:53Z

Updated by mfriedrich on 2016-01-22 15:58:53 00:00

Relates set to 11022

icinga-migration · 2016-01-25T10:34:21Z

Updated by mfriedrich on 2016-01-25 10:34:21 00:00

Target Version changed from 2.5.0 to 2.4.2

icinga-migration · 2016-01-27T09:34:50Z

Updated by mfriedrich on 2016-01-27 09:34:50 00:00

Assigned to changed from gbeutner to mfriedrich

icinga-migration · 2016-01-28T14:48:37Z

Updated by mfriedrich on 2016-01-28 14:48:37 00:00

Relates deleted ~~11022~~

icinga-migration · 2016-01-28T14:48:50Z

Updated by mfriedrich on 2016-01-28 14:48:50 00:00

Duplicated set to 11022

icinga-migration · 2016-02-03T09:05:04Z

Updated by Anonymous on 2016-02-03 09:05:04 00:00

Status changed from Assigned to Resolved

Applied in changeset 2dc385e.

icinga-migration · 2016-02-03T11:38:16Z

Updated by mfriedrich on 2016-02-03 11:38:16 00:00

Status changed from Resolved to Assigned

There's still an existing reference to the HttpServerConnection object.

Tests

for i in {1..100000};do curl -k https://root:icinga@localhost:5665/v1/objects/hosts >/dev/null 2>&1; done

watch -n1 ps -o nlwp `pidof icinga2`

Debug

[2016-02-03 10:36:23  0000] information/HttpServerConnection: Request: GET /v1/objects/hosts (root)
[New Thread 0x7fffda780700 (LWP 23665)]
[Thread 0x7fffdaa4b700 (LWP 23649) exited]
Detaching after fork from child process 23666.
[Switching to Thread 0x7fffdaa0a700 (LWP 23650)]

Breakpoint 2, icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=4) at /root/coding/icinga2/lib/base/tlsstream.cpp:132
132     boost::mutex::scoped_lock lock(m_Mutex);
(gdb) bt

#0  icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=4) at /root/coding/icinga2/lib/base/tlsstream.cpp:132
#1  0x00007ffff70aa535 in icinga::SocketEvents::ThreadProc (tid=0) at /root/coding/icinga2/lib/base/socketevents.cpp:159
#2  0x00007ffff7093f67 in boost::_bi::list1 >::operator() (this=0x7fffe8004f30, f=@0x7fffe8004f28: 0x7ffff70a9d56 , a=...)

at /usr/include/boost/bind/bind.hpp:253

#3  0x00007ffff7093f1b in boost::_bi::bind_t > >::operator() (this=0x7fffe8004f28) at /usr/include/boost/bind/bind_template.hpp:20
#4  0x00007ffff7093ee0 in boost::detail::thread_data > > >::run (this=0x7fffe8004d70) at /usr/include/boost/thread/detail/thread.hpp:117
#5  0x00007ffff7bd124a in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:165
#6  0x00007ffff3dd1dc5 in start_thread (arg=0x7fffdaa0a700) at pthread_create.c:308
#7  0x00007ffff40dc21d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

(gdb) c
Continuing.
Detaching after fork from child process 23667.
[2016-02-03 10:36:37 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!ping6' (PID: 23664, arguments: '/usr/lib/nagios/plugins/check_ping' '-6' '-H' '::1' '-c' '200,15' '-w' '100,5') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_ping) failed: No such file or directory

Breakpoint 2, icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=1) at /root/coding/icinga2/lib/base/tlsstream.cpp:132
132 boost::mutex::scoped_lock lock(m_Mutex);
(gdb) bt

#0  icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=1) at /root/coding/icinga2/lib/base/tlsstream.cpp:132
#1  0x00007ffff70aa535 in icinga::SocketEvents::ThreadProc (tid=0) at /root/coding/icinga2/lib/base/socketevents.cpp:159
#2  0x00007ffff7093f67 in boost::_bi::list1 >::operator() (this=0x7fffe8004f30, f=@0x7fffe8004f28: 0x7ffff70a9d56 , a=...)

at /usr/include/boost/bind/bind.hpp:253

#3  0x00007ffff7093f1b in boost::_bi::bind_t > >::operator() (this=0x7fffe8004f28) at /usr/include/boost/bind/bind_template.hpp:20
#4  0x00007ffff7093ee0 in boost::detail::thread_data > > >::run (this=0x7fffe8004d70) at /usr/include/boost/thread/detail/thread.hpp:117
#5  0x00007ffff7bd124a in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:165
#6  0x00007ffff3dd1dc5 in start_thread (arg=0x7fffdaa0a700) at pthread_create.c:308
#7  0x00007ffff40dc21d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

(gdb) c
Continuing.
[New Thread 0x7fffdaa4b700 (LWP 23668)]
Detaching after fork from child process 23669.
[2016-02-03 10:36:37 0000] information/Application: We jumped forward in time: 26.6704 seconds
[2016-02-03 10:36:43 0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:6.00955s, minor_faults:8, major_faults:0, voluntary_csw:7, involuntary_csw:1
[2016-02-03 10:36:43 0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:12.5225s, minor_faults:8, major_faults:0, voluntary_csw:28, involuntary_csw:0

Breakpoint 1, icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=1) at /root/coding/icinga2/lib/base/tlsstream.cpp:209
209 Close();
(gdb) info b
Num Type Disp Enb Address What
1 breakpoint keep y 0x00007ffff70db2a3 in icinga::TlsStream::OnEvent(int) at /root/coding/icinga2/lib/base/tlsstream.cpp:209
breakpoint already hit 1 time
2 breakpoint keep y 0x00007ffff70daf37 in icinga::TlsStream::OnEvent(int) at /root/coding/icinga2/lib/base/tlsstream.cpp:132
breakpoint already hit 6 times
3 breakpoint keep y 0x00007ffff70b6c0e in icinga::Stream::Close() at /root/coding/icinga2/lib/base/stream.cpp:81
breakpoint already hit 6 times
4 breakpoint keep y 0x00007ffff60a9ecb in icinga::HttpServerConnection::DataAvailableHandler() at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:199
breakpoint already hit 3 times
(gdb) s
icinga::TlsStream::Close (this=0x7fffe8004bc0) at /root/coding/icinga2/lib/base/tlsstream.cpp:324
324 CloseInternal(false);
(gdb) s
icinga::TlsStream::CloseInternal (this=0x7fffe8004bc0, inDestructor=false) at /root/coding/icinga2/lib/base/tlsstream.cpp:329
329 if (m_Eof)
(gdb) l
324 CloseInternal(false);
325 }
326
327 void TlsStream::CloseInternal(bool inDestructor)
328 {
329 if (m_Eof)
330 return;
331
332 m_Eof = true;
333
(gdb) n
332 m_Eof = true;
(gdb)
334 if (!inDestructor)
(gdb)
335 SignalDataAvailable();
(gdb)

Breakpoint 4, icinga::HttpServerConnection::DataAvailableHandler (this=0x7fffe8008470) at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:199
199 bool close = false;
(gdb) p m_Eof
No symbol "m_Eof" in current context.
(gdb) up

#1  0x00007ffff60ac480 in boost::_mfi::mf0::call > (this=0x7fffe8004af0, u=(boost::intrusive_ptr) 0x7fffe8008470)

at /usr/include/boost/bind/mem_fn_template.hpp:40
40 BOOST_MEM_FN_RETURN (get_pointer(u)->*f_)();
(gdb) down

#0  icinga::HttpServerConnection::DataAvailableHandler (this=0x7fffe8008470) at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:199

199 bool close = false;
(gdb) p *m_Stream.px
$10 = {
icinga::Stream = {
icinga::Object = {
_vptr.Object = 0x7ffff74450b0 <vtable for icinga::TlsStream 16>,
static TypeInstance = (boost::intrusive_ptricinga::Type) 0x65cc50,
m_References = 4,
m_Mutex = 0,
m_LockOwner = 0
},
members of icinga::Stream:
OnDataAvailable = ,
m_Mutex = {
m =
},
m_CV = {
internal_mutex = ,
cond = {
__data = {
__lock = 0,
__futex = 2,
__total_seq = 1,
__wakeup_seq = 1,
__woken_seq = 1,
__mutex = 0x7fffe8004c20,
__nwaiters = 0,
__broadcast_seq = 1
},
__size = "\000\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000 L\000\350\377\177\000\000\000\000\000\000\001\000\000",
__align = 8589934592
}
}
},
icinga::SocketEvents = {
_vptr.SocketEvents = 0x7ffff7445160 <vtable for icinga::TlsStream 192>,
m_ID = 0,
m_FD = 13,
m_Events = true,
m_PFD = 0x7fffe4000d50,
static m_NextID = 1
},
members of icinga::TlsStream:
m_SSL = (boost::shared_ptr<ssl_st>) (count 1, weak count 1) 0x7fffe8007710,
m_Eof = true,
--~~Type to continue, or q to quit--~~
m_Mutex = {
m =
},
m_CV = {
internal_mutex = ,
cond = {
__data = {
__lock = 0,
__futex = 2,
__total_seq = 1,
__wakeup_seq = 1,
__woken_seq = 1,
__mutex = 0x7fffe8004cd8,
__nwaiters = 0,
__broadcast_seq = 1
},
__size = "\000\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\330L\000\350\377\177\000\000\000\000\000\000\001\000\000",
__align = 8589934592
}
},
m_HandshakeOK = true,
m_VerifyOK = true,
m_ErrorCode = 0,
m_ErrorOccurred = false,
m_Socket = (boost::intrusive_ptricinga::Socket) 0x7fffc8000ab0,
m_Role = icinga::RoleServer,
m_SendQ = (boost::intrusive_ptricinga::FIFO) 0x7fffe80072d0,
m_RecvQ = (boost::intrusive_ptricinga::FIFO) 0x7fffe80074f0,
m_CurrentAction = icinga::TlsActionRead,
m_Retry = true,
m_Shutdown = false,
static m_SSLIndex = 0,
static m_SSLIndexInitialized = true
}
(gdb) l
194 m_PendingRequests--;
195 }
196
197 void HttpServerConnection::DataAvailableHandler(void)
198 {
199 bool close = false;
200
201 if (!m_Stream->IsEof()) {
202 boost::mutex::scoped_lock lock(m_DataHandlerMutex);
203
(gdb) n
201 if (!m_Stream->IsEof()) {
(gdb)
[New Thread 0x7fffda73f700 (LWP 23670)]
Detaching after fork from child process 23671.
[2016-02-03 10:38:39 0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:122.298s, minor_faults:72, major_faults:0, voluntary_csw:23, involuntary_csw:4
214 close = true;
(gdb) l
209 << "Error while reading Http request: " << DiagnosticInformation(ex);
210
211 close = true;
212 }
213 } else
214 close = true;
215
216 if (close)
217 Disconnect();
218 }
(gdb) n
216 if (close)
(gdb) p close
$11 = true
(gdb) n
217 Disconnect();
(gdb) s
[New Thread 0x7fffda6fe700 (LWP 23672)]
Detaching after fork from child process 23673.
icinga::HttpServerConnection::Disconnect (this=0x7fffe8008470) at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:76
76 Log(LogDebug, "HttpServerConnection", "Http client disconnected");
(gdb) l
71 return m_Stream;
72 }
73
74 void HttpServerConnection::Disconnect(void)
75 {
76 Log(LogDebug, "HttpServerConnection", "Http client disconnected");
77
78 ApiListener::Ptr listener = ApiListener::GetInstance();
79 listener->RemoveHttpClient(this);
80
(gdb) n
Detaching after fork from child process 23674.
[2016-02-03 10:38:39 0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:86.7289s, minor_faults:3, major_faults:0, voluntary_csw:13, involuntary_csw:0
[2016-02-03 10:39:04 0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:147.059s, minor_faults:50, major_faults:0, voluntary_csw:28, involuntary_csw:0
[2016-02-03 10:38:39 0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:20.1464s, minor_faults:96, major_faults:0, voluntary_csw:32, involuntary_csw:3
[2016-02-03 10:38:39 0000] information/Application: We jumped forward in time: 35.5747 seconds
[2016-02-03 10:39:04 0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:141.05s, minor_faults:89, major_faults:0, voluntary_csw:15, involuntary_csw:1
[2016-02-03 10:39:04 0000] warning/Process: Killing process group 23671 ('/usr/lib/nagios/plugins/check_http' '-I' '127.0.0.1' '-u' '/') after timeout of 60 seconds
[2016-02-03 10:38:54 0000] warning/Process: Killing process group 23667 ('/usr/lib/nagios/plugins/check_ssh' '127.0.0.1') after timeout of 60 seconds
[2016-02-03 10:39:04 0000] information/HttpServerConnection: No messages for Http connection have been received in the last 10 seconds.
[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!http' (PID: 23671, arguments: '/usr/lib/nagios/plugins/check_http' '-I' '127.0.0.1' '-u' '/') terminated with exit code 128, output:
[New Thread 0x7fffda6bd700 (LWP 23675)]
Detaching after fork from child process 23676.
[2016-02-03 10:39:04 0000] warning/Process: Killing process group 23669 ('/usr/lib/nagios/plugins/check_ping' '-4' '-H' '127.0.0.1' '-c' '200,15' '-w' '100,5') after timeout of 60 seconds
[New Thread 0x7fffda67c700 (LWP 23677)]
Detaching after fork from child process 23678.
[2016-02-03 10:39:04 0000] warning/ThreadPool: Event call took user:0.01s, system:0s, wait:161.154s, minor_faults:80, major_faults:0, voluntary_csw:44, involuntary_csw:2
[2016-02-03 10:39:04 0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:9.70591s, minor_faults:120, major_faults:0, voluntary_csw:17, involuntary_csw:0
[2016-02-03 10:39:04 0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:24.8513s, minor_faults:73, major_faults:0, voluntary_csw:21, involuntary_csw:1
[New Thread 0x7fffda63b700 (LWP 23679)]
Detaching after fork from child process 23680.
[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!disk' (PID: 23666, arguments: '/usr/lib/nagios/plugins/check_disk' '-c' '10' '-w' '20' '-K' '10' '-W' '20' '-X' 'none' '-X' 'tmpfs' '-X' 'sysfs' '-X' 'proc' '-X' 'devtmpfs' '-X' 'devfs' '-X' 'mtmfs' '-m') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_disk) failed: No such file or directory

[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!load' (PID: 23660, arguments: '/usr/lib/nagios/plugins/check_load' '-c' '10,6,4' '-w' '5,4,3') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_load) failed: No such file or directory

[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!ssh' (PID: 23667, arguments: '/usr/lib/nagios/plugins/check_ssh' '127.0.0.1') terminated with exit code 128, output:
[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!ping4' (PID: 23669, arguments: '/usr/lib/nagios/plugins/check_ping' '-4' '-H' '127.0.0.1' '-c' '200,15' '-w' '100,5') terminated with exit code 128, output:
[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!swap' (PID: 23678, arguments: '/usr/lib/nagios/plugins/check_swap' '-c' '25' '-w' '50') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_swap) failed: No such file or directory

[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!disk /' (PID: 23673, arguments: '/usr/lib/nagios/plugins/check_disk' '-c' '10' '-w' '20' '-K' '10' '-W' '20' '-X' 'none' '-X' 'tmpfs' '-X' 'sysfs' '-X' 'proc' '-X' 'devtmpfs' '-X' 'devfs' '-X' 'mtmfs' '-m' '-p' '/') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_disk) failed: No such file or directory

[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!users' (PID: 23680, arguments: '/usr/lib/nagios/plugins/check_users' '-c' '50' '-w' '20') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_users) failed: No such file or directory

Detaching after fork from child process 23681.
Detaching after fork from child process 23682.
[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!procs' (PID: 23674, arguments: '/usr/lib/nagios/plugins/check_procs' '-c' '400' '-w' '250') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_procs) failed: No such file or directory

[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b' (PID: 23676, arguments: '/usr/lib/nagios/plugins/check_ping' '-H' '127.0.0.1' '-c' '5000,100' '-w' '3000,80') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_ping) failed: No such file or directory

[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!ping6' (PID: 23681, arguments: '/usr/lib/nagios/plugins/check_ping' '-6' '-H' '::1' '-c' '200,15' '-w' '100,5') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_ping) failed: No such file or directory

[2016-02-03 10:39:04 0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!ping4' (PID: 23682, arguments: '/usr/lib/nagios/plugins/check_ping' '-4' '-H' '127.0.0.1' '-c' '200,15' '-w' '100,5') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_ping) failed: No such file or directory

78 ApiListener::Ptr listener = ApiListener::GetInstance();
(gdb)
79 listener->RemoveHttpClient(this);
(gdb) p *this
$12 = {
icinga::Object = {
_vptr.Object = 0x7ffff6439d90 <vtable for icinga::HttpServerConnection 16>,
static TypeInstance = (boost::intrusive_ptricinga::Type) 0x65cc50,
m_References = 1,
m_Mutex = 0,
m_LockOwner = 0
},
members of icinga::HttpServerConnection:
m_ApiUser = (boost::intrusive_ptricinga::ApiUser) 0x0,
m_Stream = (boost::intrusive_ptricinga::TlsStream) 0x7fffe8004bc0,
m_Seen = 1454495783.131804,
m_CurrentRequest = {
Complete = false,
RequestMethod = "",
RequestUrl = (boost::intrusive_ptricinga::Url) 0x0,
ProtocolVersion = icinga::HttpVersion11,
Headers = (boost::intrusive_ptricinga::Dictionary) 0x7fffe4001ff0,
m_Stream = (boost::intrusive_ptricinga::Stream) 0x7fffe8004bc0,
m_ChunkContext = (boost::shared_ptricinga::ChunkReadContext) 0x0,
m_State = icinga::HttpRequestStart,
m_Body = (boost::intrusive_ptricinga::FIFO) 0x0
},
m_DataHandlerMutex = {
m =
},
m_RequestQueue = {
m_ID = 6,
static m_NextID = 7,
m_ThreadCount = 1,
m_Spawned = true,
m_Mutex = {
m =
},
m_CVEmpty = {
internal_mutex = ,
cond = {
__data = {
__lock = 0,
__futex = 1,
__total_seq = 1,
__wakeup_seq = 0,
__woken_seq = 0,
---Type to continue, or q to quit---q
Quit
(gdb) l
74 void HttpServerConnection::Disconnect(void)
75 {
76 Log(LogDebug, "HttpServerConnection", "Http client disconnected");
77
78 ApiListener::Ptr listener = ApiListener::GetInstance();
79 listener->RemoveHttpClient(this);
80
81 m_Stream->Shutdown();
82 }
83
(gdb) n
81 m_Stream->Shutdown();
(gdb) p *m_Stream.px
$13 = {
icinga::Stream = {
icinga::Object = {
_vptr.Object = 0x7ffff74450b0 <vtable for icinga::TlsStream 16>,
static TypeInstance = (boost::intrusive_ptricinga::Type) 0x65cc50,
m_References = 4,
m_Mutex = 0,
m_LockOwner = 0
},
members of icinga::Stream:
OnDataAvailable = ,
m_Mutex = {
m =
},
m_CV = {
internal_mutex = ,
cond = {
__data = {
__lock = 0,
__futex = 2,
__total_seq = 1,
__wakeup_seq = 1,
__woken_seq = 1,
__mutex = 0x7fffe8004c20,
__nwaiters = 0,
__broadcast_seq = 1
},
__size = "\000\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000 L\000\350\377\177\000\000\000\000\000\000\001\000\000",
__align = 8589934592
}
}
},
icinga::SocketEvents = {
_vptr.SocketEvents = 0x7ffff7445160 <vtable for icinga::TlsStream 192>,
m_ID = 0,
m_FD = 13,
m_Events = true,
m_PFD = 0x7fffe4000d50,
static m_NextID = 1
},
members of icinga::TlsStream:
m_SSL = (boost::shared_ptr<ssl_st>) (count 1, weak count 1) 0x7fffe8007710,
m_Eof = true,
---Type to continue, or q to quit---q
Quit
(gdb) l
76 Log(LogDebug, "HttpServerConnection", "Http client disconnected");
77
78 ApiListener::Ptr listener = ApiListener::GetInstance();
79 listener->RemoveHttpClient(this);
80
81 m_Stream->Shutdown();
82 }
83
84 bool HttpServerConnection::ProcessMessage(void)
85 {
(gdb) n
82 }
(gdb) bt

#0  icinga::HttpServerConnection::Disconnect (this=0x7fffe8008470) at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:82
#1  0x00007ffff60a9f59 in icinga::HttpServerConnection::DataAvailableHandler (this=0x7fffe8008470) at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:217
#2  0x00007ffff60ac480 in boost::_mfi::mf0::call > (this=0x7fffe8004af0, u=(boost::intrusive_ptr) 0x7fffe8008470)

at /usr/include/boost/bind/mem_fn_template.hpp:40

#3  0x00007ffff60ac305 in boost::_mfi::mf0::operator() > (this=0x7fffe8004af0, u=(boost::intrusive_ptr) 0x7fffe8008470)

at /usr/include/boost/bind/mem_fn_template.hpp:55

#4  0x00007ffff60abe52 in boost::_bi::list1 > >::operator(), boost::_bi::list1 const&> > (this=0x7fffe8004b00, f=..., a=...) at /usr/include/boost/bind/bind.hpp:253
#5  0x00007ffff60abbbe in boost::_bi::bind_t, boost::_bi::list1 > > >::operator() > (this=0x7fffe8004af0, a1=(boost::intrusive_ptr) 0x7fffe8004bc0) at /usr/include/boost/bind/bind_template.hpp:47
#6  0x00007ffff60ab994 in boost::detail::function::void_function_obj_invoker1, boost::_bi::list1 > > >, void, boost::intrusive_ptr const&>::invoke (function_obj_ptr=..., a0=(boost::intrusive_ptr) 0x7fffe8004bc0) at /usr/include/boost/function/function_template.hpp:153
#7  0x00007ffff70bb4b9 in boost::function1 const&>::operator() (this=0x7fffe8004ae8, a0=(boost::intrusive_ptr) 0x7fffe8004bc0)

at /usr/include/boost/function/function_template.hpp:767

#8  0x00007ffff70bb337 in boost::signals2::detail::call_with_tuple_args::m_invoke const&)>, 0u, boost::intrusive_ptr const&>(void*, boost::function const&)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::tuple const&>) const (this=0x7fffda9f8d1f, func=...,

args=std::tuple containing = {...}) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:89

#9  0x00007ffff70bb1e2 in boost::signals2::detail::call_with_tuple_args::operator() const&)>, boost::intrusive_ptr const&>(boost::function const&)>&, std::tuple const&>) const (this=0x7fffda9f8d1f, func=..., args=std::tuple containing = {...})

at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:78

#10 0x00007ffff70bb0a0 in boost::signals2::detail::variadic_slot_invoker const&>::m_invoke >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >(boost::shared_ptr >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > const&, boost::signals2::detail::void_type const*) const (this=0x7fffda9f8ff0, connectionBody=warning: RTTI symbol not found for class 'boost::detail::sp_counted_impl_p >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >'

(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional >, boost::signals2::slot<void(const boost::intrusive_ptricinga::Stream&), boost::function<void(const boost::intrusive_ptricinga::Stream&)> >, boost::signals2::mutex> >) (count 1, weak count 1) 0x7fffe8004ab0) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:114

#11 0x00007ffff70bad98 in boost::signals2::detail::variadic_slot_invoker const&>::operator() >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >(boost::shared_ptr >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > const&) const (this=0x7fffda9f8ff0, connectionBody=warning: RTTI symbol not found for class 'boost::detail::sp_counted_impl_p >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >'

(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional >, boost::signals2::slot<void(const boost::intrusive_ptricinga::Stream&), boost::function<void(const boost::intrusive_ptricinga::Stream&)> >, boost::signals2::mutex> >) (count 1, weak count 1) 0x7fffe8004ab0) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:107

#12 0x00007ffff70ba2af in boost::signals2::detail::slot_call_iterator_t const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >::dereference() const (this=0x7fffda9f8e30) at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
#13 0x00007ffff70b9d16 in boost::iterator_core_access::dereference const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > const&) (f=...)

--~~Type to continue, or q to quit--~~
at /usr/include/boost/iterator/iterator_facade.hpp:514

#14 0x00007ffff70b9740 in boost::iterator_facade const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::single_pass_traversal_tag, boost::signals2::detail::void_type const&, long>::operator*() const (this=0x7fffda9f8e30) at /usr/include/boost/iterator/iterator_facade.hpp:639
#15 0x00007ffff70b8a41 in boost::signals2::optional_last_value::operator() const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >) const (this=0x7fffe8000e10, first=..., last=...) at /usr/include/boost/signals2/optional_last_value.hpp:55
#16 0x00007ffff70b81c3 in boost::signals2::detail::combiner_invoker::operator(), boost::signals2::detail::slot_call_iterator_t const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value&, boost::signals2::detail::slot_call_iterator_t const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t const&>, std::_List_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >) const (this=0x7fffda9f90af, combiner=..., first=..., last=...)

at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64

#17 0x00007ffff70b7a43 in boost::signals2::detail::signal_impl const&), boost::signals2::optional_last_value, int, std::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator()(boost::intrusive_ptr const&) (this=0x7fffe80035a0,

args#0=(boost::intrusive_ptricinga::Stream) 0x7fffe8004bc0) at /usr/include/boost/signals2/detail/signal_template.hpp:246

#18 0x00007ffff70b755b in boost::signals2::signal const&), boost::signals2::optional_last_value, int, std::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator()(boost::intrusive_ptr const&) (this=0x7fffe8004be0,

args#0=(boost::intrusive_ptricinga::Stream) 0x7fffe8004bc0) at /usr/include/boost/signals2/detail/signal_template.hpp:695

#19 0x00007ffff70b69a8 in icinga::Stream::SignalDataAvailable (this=0x7fffe8004bc0) at /root/coding/icinga2/lib/base/stream.cpp:55
#20 0x00007ffff70dbbbc in icinga::TlsStream::CloseInternal (this=0x7fffe8004bc0, inDestructor=false) at /root/coding/icinga2/lib/base/tlsstream.cpp:335
#21 0x00007ffff70dbb71 in icinga::TlsStream::Close (this=0x7fffe8004bc0) at /root/coding/icinga2/lib/base/tlsstream.cpp:324
#22 0x00007ffff70db2c0 in icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=1) at /root/coding/icinga2/lib/base/tlsstream.cpp:209
#23 0x00007ffff70aa535 in icinga::SocketEvents::ThreadProc (tid=0) at /root/coding/icinga2/lib/base/socketevents.cpp:159
#24 0x00007ffff7093f67 in boost::_bi::list1 >::operator() (this=0x7fffe8004f30, f=@0x7fffe8004f28: 0x7ffff70a9d56 , a=...)

at /usr/include/boost/bind/bind.hpp:253

#25 0x00007ffff7093f1b in boost::_bi::bind_t > >::operator() (this=0x7fffe8004f28) at /usr/include/boost/bind/bind_template.hpp:20

--~~Type to continue, or q to quit--~~

#26 0x00007ffff7093ee0 in boost::detail::thread_data > > >::run (this=0x7fffe8004d70) at /usr/include/boost/thread/detail/thread.hpp:117
#27 0x00007ffff7bd124a in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:165
#28 0x00007ffff3dd1dc5 in start_thread (arg=0x7fffdaa0a700) at pthread_create.c:308
#29 0x00007ffff40dc21d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

(gdb) frame 20

#20 0x00007ffff70dbbbc in icinga::TlsStream::CloseInternal (this=0x7fffe8004bc0, inDestructor=false) at /root/coding/icinga2/lib/base/tlsstream.cpp:335

335 SignalDataAvailable();
(gdb) l
330 return;
331
332 m_Eof = true;
333
334 if (!inDestructor)
335 SignalDataAvailable();
336
337 SocketEvents::Unregister();
338
339 Stream::Close();
(gdb) fr 1

#1  0x00007ffff60a9f59 in icinga::HttpServerConnection::DataAvailableHandler (this=0x7fffe8008470) at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:217

217 Disconnect();
(gdb) fr 20

#20 0x00007ffff70dbbbc in icinga::TlsStream::CloseInternal (this=0x7fffe8004bc0, inDestructor=false) at /root/coding/icinga2/lib/base/tlsstream.cpp:335

335 SignalDataAvailable();
(gdb) l
330 return;
331
332 m_Eof = true;
333
334 if (!inDestructor)
335 SignalDataAvailable();
336
337 SocketEvents::Unregister();
338
339 Stream::Close();
(gdb) b 337
Breakpoint 5 at 0x7ffff70dbbbc: file /root/coding/icinga2/lib/base/tlsstream.cpp, line 337.
(gdb) c
Continuing.

Breakpoint 5, icinga::TlsStream::CloseInternal (this=0x7fffe8004bc0, inDestructor=false) at /root/coding/icinga2/lib/base/tlsstream.cpp:337
337 SocketEvents::Unregister();
(gdb) p (TlsStream *) 0x7fffe8004bc0
$14 = (void ()(icinga::TlsStream const, const icinga::Socket::Ptr &, const icinga::String &, icinga::ConnectionRole, const boost::shared_ptr<ssl_ctx_st> &)) 0x7fffe8004bc0
(gdb) p (TlsStream) 0x7fffe8004bc0
$15 = {void (icinga::TlsStream * const, const icinga::Socket::Ptr &, const icinga::String &, icinga::ConnectionRole, const boost::shared_ptr<ssl_ctx_st> &)} 0x7fffe8004bc0
(gdb) p ((icinga::TlsStream) 0x7fffe8004bc0).m_References
$16 = 3
(gdb) p ((icinga::HttpServerConnection) 0x7fffe8008470).m_References
$17 = 1
(gdb) bt

#0  icinga::TlsStream::CloseInternal (this=0x7fffe8004bc0, inDestructor=false) at /root/coding/icinga2/lib/base/tlsstream.cpp:337
#1  0x00007ffff70dbb71 in icinga::TlsStream::Close (this=0x7fffe8004bc0) at /root/coding/icinga2/lib/base/tlsstream.cpp:324
#2  0x00007ffff70db2c0 in icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=1) at /root/coding/icinga2/lib/base/tlsstream.cpp:209
#3  0x00007ffff70aa535 in icinga::SocketEvents::ThreadProc (tid=0) at /root/coding/icinga2/lib/base/socketevents.cpp:159
#4  0x00007ffff7093f67 in boost::_bi::list1 >::operator() (this=0x7fffe8004f30, f=@0x7fffe8004f28: 0x7ffff70a9d56 , a=...)

at /usr/include/boost/bind/bind.hpp:253

#5  0x00007ffff7093f1b in boost::_bi::bind_t > >::operator() (this=0x7fffe8004f28) at /usr/include/boost/bind/bind_template.hpp:20
#6  0x00007ffff7093ee0 in boost::detail::thread_data > > >::run (this=0x7fffe8004d70) at /usr/include/boost/thread/detail/thread.hpp:117
#7  0x00007ffff7bd124a in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:165
#8  0x00007ffff3dd1dc5 in start_thread (arg=0x7fffdaa0a700) at pthread_create.c:308
#9  0x00007ffff40dc21d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

(gdb) l
332 m_Eof = true;
333
334 if (!inDestructor)
335 SignalDataAvailable();
336
337 SocketEvents::Unregister();
338
339 Stream::Close();
340
341 boost::mutex::scoped_lock lock(m_Mutex);
(gdb) n
339 Stream::Close();
(gdb) p ((icinga::TlsStream) 0x7fffe8004bc0).m_References
$18 = 3
(gdb) p ((icinga::HttpServerConnection) 0x7fffe8008470).m_References
$19 = 1
(gdb) s

Breakpoint 3, icinga::Stream::Close (this=0x7fffe8004bc0) at /root/coding/icinga2/lib/base/stream.cpp:81
81 OnDataAvailable.disconnect_all_slots();
(gdb) bt

#0  icinga::Stream::Close (this=0x7fffe8004bc0) at /root/coding/icinga2/lib/base/stream.cpp:81
#1  0x00007ffff70dbbda in icinga::TlsStream::CloseInternal (this=0x7fffe8004bc0, inDestructor=false) at /root/coding/icinga2/lib/base/tlsstream.cpp:339
#2  0x00007ffff70dbb71 in icinga::TlsStream::Close (this=0x7fffe8004bc0) at /root/coding/icinga2/lib/base/tlsstream.cpp:324
#3  0x00007ffff70db2c0 in icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=1) at /root/coding/icinga2/lib/base/tlsstream.cpp:209
#4  0x00007ffff70aa535 in icinga::SocketEvents::ThreadProc (tid=0) at /root/coding/icinga2/lib/base/socketevents.cpp:159
#5  0x00007ffff7093f67 in boost::_bi::list1 >::operator() (this=0x7fffe8004f30, f=@0x7fffe8004f28: 0x7ffff70a9d56 , a=...)

at /usr/include/boost/bind/bind.hpp:253

#6  0x00007ffff7093f1b in boost::_bi::bind_t > >::operator() (this=0x7fffe8004f28) at /usr/include/boost/bind/bind_template.hpp:20
#7  0x00007ffff7093ee0 in boost::detail::thread_data > > >::run (this=0x7fffe8004d70) at /usr/include/boost/thread/detail/thread.hpp:117
#8  0x00007ffff7bd124a in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:165
#9  0x00007ffff3dd1dc5 in start_thread (arg=0x7fffdaa0a700) at pthread_create.c:308
#10 0x00007ffff40dc21d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

(gdb) n
82 }
(gdb) p ((icinga::HttpServerConnection) 0x7fffe8008470).m_References
$20 = 1
(gdb) p ((icinga::TlsStream) 0x7fffe8004bc0).m_References
$21 = 3

icinga-migration · 2016-02-03T11:38:23Z

Updated by mfriedrich on 2016-02-03 11:38:23 00:00

Research

RegisterDataAvailableHandler, OnDataAvailable()

The older fix works on OSX (boost 1.59 ) but not RHEL7 (boost 1.53)

[root@e4e752bf970b coding]# vim test.cpp

#include 
#include 
#include 
#include 

class Object
{
public:
        void f(void)
        { }

        int m_Refs;
};

void intrusive_ptr_add_ref(Object *object)
{
        std::cout << "add_ref" << std::endl;
        object->m_Refs  ;
}

void intrusive_ptr_release(Object *object)
{
        std::cout << "release" << std::endl;
        object->m_Refs--;
}

boost::signals2::signal s;

int main(void)
{
        boost::intrusive_ptr p = new Object();

        s.connect(boost::bind(&Object::f, p));

        p.reset();

        s();

        s.disconnect_all_slots();

//      _exit(0);
}


[root@e4e752bf970b coding]# g   -g -o test test.cpp
[root@e4e752bf970b coding]# ./test | grep release | wc -l
16
[root@e4e752bf970b coding]# ./test | grep add_ref | wc -l
17

icinga-migration · 2016-02-03T11:54:18Z

Updated by mfriedrich on 2016-02-03 11:54:18 00:00

http://stackoverflow.com/questions/2049291/force-deletion-of-slot-in-boostsignals2

icinga-migration · 2016-02-03T12:48:04Z

Updated by mfriedrich on 2016-02-03 12:48:04 00:00

Using the fix from caf3380

Every 1.0s: ps -o nlwp 10491                                                                                                                                                                                                                            Wed Feb  3 13:47:40 2016

NLWP
  27

icinga-migration · 2016-02-03T12:56:37Z

Updated by mfriedrich on 2016-02-03 12:56:37 00:00

Affects boost versions > 1.49 and < 1.59 (e.g. RHEL 7).

icinga-migration · 2016-02-03T12:56:49Z

Updated by mfriedrich on 2016-02-03 12:56:49 00:00

Status changed from Assigned to Resolved

icinga-migration · 2016-02-03T14:13:58Z

Updated by mfriedrich on 2016-02-03 14:13:58 00:00

Subject changed from API query opens the thread but the thread does not die. to _API queries cause memory leaks _

icinga-migration · 2016-02-04T09:08:35Z

Updated by mfriedrich on 2016-02-04 09:08:35 00:00

Relates set to 10840

icinga-migration · 2016-02-04T12:43:39Z

Updated by mfriedrich on 2016-02-04 12:43:39 00:00

https://svn.boost.org/trac/boost/ticket/8533
boostorg/signals2@c22312b

icinga-migration · 2016-02-08T08:59:32Z

Updated by mfriedrich on 2016-02-08 08:59:32 00:00

Duplicated set to 11108

icinga-migration · 2016-02-23T09:58:18Z

Updated by gbeutner on 2016-02-23 09:58:18 00:00

Backport? changed from Not yet backported to Already backported

icinga-migration closed this as completed Feb 3, 2016

icinga-migration added the blocker Blocks a release or needs immediate attention label Jan 17, 2017

icinga-migration added bug Something isn't working area/api REST API labels Jan 17, 2017

icinga-migration added this to the 2.4.2 milestone Jan 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dev.icinga.com #10655] API queries cause memory leaks #3677

[dev.icinga.com #10655] API queries cause memory leaks #3677

icinga-migration commented Nov 17, 2015

icinga-migration commented Nov 25, 2015

icinga-migration commented Nov 28, 2015

icinga-migration commented Nov 30, 2015

icinga-migration commented Dec 15, 2015

icinga-migration commented Dec 18, 2015

icinga-migration commented Jan 7, 2016

icinga-migration commented Jan 11, 2016

icinga-migration commented Jan 13, 2016

icinga-migration commented Jan 13, 2016

icinga-migration commented Jan 13, 2016

icinga-migration commented Jan 13, 2016

icinga-migration commented Jan 22, 2016

icinga-migration commented Jan 25, 2016

icinga-migration commented Jan 27, 2016

icinga-migration commented Jan 28, 2016

icinga-migration commented Jan 28, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 4, 2016

icinga-migration commented Feb 4, 2016

icinga-migration commented Feb 8, 2016

icinga-migration commented Feb 23, 2016

[dev.icinga.com #10655] API queries cause memory leaks #3677

[dev.icinga.com #10655] API queries cause memory leaks #3677

Comments

icinga-migration commented Nov 17, 2015

icinga-migration commented Nov 25, 2015

icinga-migration commented Nov 28, 2015

icinga-migration commented Nov 30, 2015

icinga-migration commented Dec 15, 2015

icinga-migration commented Dec 18, 2015

icinga-migration commented Jan 7, 2016

icinga-migration commented Jan 11, 2016

icinga-migration commented Jan 13, 2016

icinga-migration commented Jan 13, 2016

icinga-migration commented Jan 13, 2016

icinga-migration commented Jan 13, 2016

icinga-migration commented Jan 22, 2016

icinga-migration commented Jan 25, 2016

icinga-migration commented Jan 27, 2016

icinga-migration commented Jan 28, 2016

icinga-migration commented Jan 28, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

Tests

Debug

icinga-migration commented Feb 3, 2016

Research

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 3, 2016

icinga-migration commented Feb 4, 2016

icinga-migration commented Feb 4, 2016

icinga-migration commented Feb 8, 2016

icinga-migration commented Feb 23, 2016