[dev.icinga.com #10495] Incorrect JSON-RPC message causes Icinga 2 to crash #3569

icinga-migration · 2015-10-30T09:32:33Z

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10495

Created by netphantm on 2015-10-30 09:32:33 +00:00

Assignee: gbeutner
Status: Resolved (closed on 2015-11-02 16:50:03 +00:00)
Target Version: 2.4.0
Last Update: 2015-11-12 15:34:44 +00:00 (in Redmine)

Icinga Version: 2.3.11 debian 8
Backport?: No
Include in Changelog: 1

I tried to acknowledge a check with curl and each time I sent the POST with curl, the icinga2 daemon crashes with this attached report.
the command I try is this:
curl --tlsv1 --cacert /var/lib/icinga2/ca/ca.crt --key /etc/icinga2/pki/de-nue-monitoring-staging.censhare.de.key --cert /etc/icinga2/pki/de-nue-monitoring-staging.censhare.de.crt --anyauth -v -u icingaadmin:pass4icinga -k -s -X POST "https://localhost:5665/v1/actions/acknowledge-problem?type=Service&filter=service.name"ping4"&filter=host.name"sn-svc-vm-02"" -d "{\author\. D. Salinger\",\comment\ thought what I"d do was I"d pretend I was one of those deaf-mutes\",\notify\ }"
I"m not sure if it"s the right command or even a bug, didn"t find to much examples out there on how to acknowledge through the API. without specifying the cert, ca and key, I coudn"t get any results from the API. here is the output:

Hostname was NOT found in DNS cache
Trying ::1...
connect to ::1 port 5665 failed: Connection refused
Trying 127.0.0.1...
Connected to localhost (127.0.0.1) port 5665 (#0)
successfully set certificate verify locations:
CAfile: /var/lib/icinga2/ca/ca.crt
CApath: /etc/ssl/certs
SSLv3, TLS handshake, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Request CERT (13):
SSLv3, TLS handshake, Server finished (14):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Client key exchange (16):
SSLv3, TLS handshake, CERT verify (15):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSL connection using TLSv1.0 / AES256-SHA
Server certificate:
subject: CN=de-nue-monitoring-staging.censhare.de
start date: 2015-10-28 15:44:35 GMT
expire date: 2030-10-24 15:44:35 GMT
issuer: CN=Icinga CA
SSL certificate verify ok.
> POST /v1/actions/acknowledge-problem?type=Service&filter=service.name"ping4"&filter=host.name"sn-svc-vm-02" HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:5665
> Accept: /
> Content-Length: 122
> Content-Type: application/x-www-form-urlencoded
>
upload completely sent off: 122 out of 122 bytes
SSLv3, TLS alert, Client hello (1):
Empty reply from server
Connection #0 to host localhost left intact

if you need more info, please ask.

Attachments

report.1446049105.5589809 netphantm - 2015-10-30 09:25:57 +00:00
backtrace netphantm - 2015-11-01 11:52:33 +00:00

Changesets

2015-11-02 16:45:44 +00:00 by (unknown) 9c57589

Fix crash in JsonRpcClient::DataAvailableHandler

fixes #10495

The text was updated successfully, but these errors were encountered:

icinga-migration · 2015-10-30T11:21:24Z

Updated by jflach on 2015-10-30 11:21:24 +00:00

I was not able to reproduce this. Could you provide a debug log snippet? It looks like the crash occurs when the connection is shut down, I would like to confirm this.

icinga-migration · 2015-10-30T13:33:08Z

Updated by mfriedrich on 2015-10-30 13:33:08 +00:00

Obviously the example needs to be updated. It"s in a different format compared to the other commands.

icinga-migration · 2015-10-30T13:35:15Z

Updated by mfriedrich on 2015-10-30 13:35:15 +00:00

Status changed from New to Feedback
Assigned to set to netphantm

Please run icinga2 with gdb/lldb (details in the documentation) and retry your curl request. Then generate a backtrace and attach the backtrace to this ticket.

icinga-migration · 2015-11-01T11:52:50Z

Updated by netphantm on 2015-11-01 11:52:50 +00:00

File added backtrace

backtrace attached

icinga-migration · 2015-11-02T16:32:00Z

Updated by gbeutner on 2015-11-02 16:32:00 +00:00

I can reproduce this here.

Connect with openssl s_client
Send a random digit, hit enter a few times
Icinga crashes

(lldb) bt
- thread Further differentiation between service and systemctl #8: tid = 0x2cf18d, 0x00007fff85a320ae libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGABRT
  - frame #0: 0x00007fff85a320ae libsystem_kernel.dylib__pthread_kill + 10 frame #1: 0x00007fff83487500 libsystem_pthread.dylibpthread_kill + 90
    frame Add notes about certificate permission #2: 0x00007fff8a53337b libsystem_c.dylibabort + 129 frame #3: 0x00007fff8a4fa9c4 libsystem_c.dylib__assert_rtn + 321
    frame Extend http CheckCommand to accept --expected http status return codes #4: 0x0000000100592ff6 libremote.dylibboost::mutex::~mutex(this=0x00000001042e8cd0) + 134 at mutex.hpp:111 frame #5: 0x00000001005924c5 libremote.dylibboost::mutex::~mutex(this=0x00000001042e8cd0) + 21 at mutex.hpp:108
    frame Added check_ide_smart command #6: 0x000000010072d837 libremote.dylibicinga::JsonRpcConnection::~JsonRpcConnection(this=0x00000001042e8c60) + 71 at jsonrpcconnection.hpp:51 frame #7: 0x000000010072c205 libremote.dylibicinga::JsonRpcConnection::~JsonRpcConnection(this=0x00000001042e8c60) + 21 at jsonrpcconnection.hpp:51
    frame Further differentiation between service and systemctl #8: 0x000000010072c229 libremote.dylibicinga::JsonRpcConnection::~JsonRpcConnection(this=0x00000001042e8c60) + 25 at jsonrpcconnection.hpp:51 frame #9: 0x000000010055ff62 libremote.dylibicinga::intrusive_ptr_release(object=0x00000001042e8c60) + 82 at object.hpp:166
    frame Fixing some config errors #10: 0x000000010061120d libremote.dylibboost::intrusive_ptr::~intrusive_ptr(this=0x00007000004143f0) + 45 at intrusive_ptr.hpp:97 frame #11: 0x00000001005bec75 libremote.dylibboost::intrusive_ptr::~intrusive_ptr(this=0x00007000004143f0) + 21 at intrusive_ptr.hpp:96
    frame Fix reading perfdata in compat/checkresultreader #12: 0x0000000100729175 libremote.dylibicinga::JsonRpcConnection::Disconnect(this=0x00000001042e8c60) + 373 at jsonrpcconnection.cpp:118 frame #13: 0x00000001007284ec libremote.dylibicinga::JsonRpcConnection::DataAvailableHandler(this=0x00000001042e8c60) + 412 at jsonrpcconnection.cpp:203
    frame Conditionally enable MySQL and PostgresSQL, add support for FreeBSD and DragonFlyBSD #14: 0x000000010072cba2 libremote.dylibboost::_mfi::mf0::operator(this=0x0000000108407830, p=0x00000001042e8c60)(icinga::JsonRpcConnection*) const + 114 at mem_fn_template.hpp:49 frame #15: 0x000000010072cb00 libremote.dylibvoid boost::_bi::list1 >::operator(this=0x0000000108407840, (null)=type @ 0x00007000004147b8, f=0x0000000108407830, a=0x00007000004147e8, (null)=0), boost::_bi::list1 const&> >(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list1 const&>&, int) + 80 at bind.hpp:255
    frame Fix Remote Client Documentation #16: 0x000000010072ca9f libremote.dylibvoid boost::_bi::bind_t, boost::_bi::list1 > >::operator(this=0x0000000108407830, a1=0x0000700000414fe0) const&>(boost::intrusive_ptr const&&&) + 79 at bind.hpp:907 frame #17: 0x000000010072c858 libremote.dylibboost::detail::function::void_function_obj_invoker1, boost::_bi::list1 > >, void, boost::intrusive_ptr const&>::invoke(function_obj_ptr=0x0000000108407830, a0=0x0000700000414fe0) + 56 at function_template.hpp:159
    frame Fix group permission in prepare-dirs #18: 0x0000000100fc7f9b libbase.dylibboost::function1 const&>::operator(this=0x0000000108407828, a0=0x0000700000414fe0)(boost::intrusive_ptr const&) const + 155 at function_template.hpp:770 frame #19: 0x0000000100fc7eeb libbase.dylibboost::signals2::detail::void_type boost::signals2::detail::call_with_tuple_args::m_invoke @ 0x00007000004148c0, args=tuple &> @ 0x00007000004148b8, (null)=0x0000000000000000)>, 0u, boost::intrusive_ptr const&>(boost::function const&)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::__1::tuple const&>, boost::enable_if const&)>::result_type>, void>::type*) const + 59 at variadic_slot_invoker.hpp:95
    frame Fix rpmspec build for el7-like machines #20: 0x0000000100fc7e7a libbase.dylibboost::signals2::detail::void_type boost::signals2::detail::call_with_tuple_args::operator(this=0x00007000004149e0, func=0x0000000108407828, args=tuple &> @ 0x0000700000414928, (null)=size_t<1> @ 0x0000700000414920) const&)>, boost::intrusive_ptr const&, 1ul>(boost::function const&)>&, std::__1::tuple const&>, mpl_::size_t<1ul>) const + 202 at variadic_slot_invoker.hpp:80 frame #21: 0x0000000100fc7d9c libbase.dylibboost::signals2::detail::void_type boost::signals2::detail::variadic_slot_invoker const&>::operator(this=0x0000700000414e80, connectionBody=0x00000001042a9f90) >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >(boost::shared_ptr >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > const&) const + 76 at variadic_slot_invoker.hpp:123
    frame Removed mail_from from check_ssmtp command. #22: 0x0000000100fc7c7b libbase.dylibboost::signals2::detail::slot_call_iterator_t const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >::dereference(this=0x0000700000414b20) const + 107 at slot_call_iterator.hpp:109 frame #23: 0x0000000100fc7be5 libbase.dylibboost::signals2::detail::slot_call_iterator_t const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >::reference boost::iterators::iterator_core_access::dereference const&>, std::__1::__list_iterator >, boost::signals2::slot const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > const&) + 21 at iterator_facade.hpp:549
    frame ITL: Implement various options for the Snmp CheckCommand, including mib-list #24: 0x0000000100fc7b1d libbase.dylibboost::iterators::detail::iterator_facade_base const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::iterators::single_pass_traversal_tag, boost::signals2::detail::void_type&, long, false, false>::operator*(this=0x0000700000414b20) const + 29 at iterator_facade.hpp:655 frame #25: 0x0000000100fc7a55 libbase.dylibvoid boost::signals2::optional_last_value::operator(this=0x00000001035b2560, first=slot_call_iterator_t &>, std::__1::__list_iterator >, boost::signals2::slot &), boost::function &)> >, boost::signals2::mutex> >, void >, boost::signals2::detail::connection_body >, boost::signals2::slot &), boost::function &)> >, boost::signals2::mutex> > @ 0x0000700000414b20, last=slot_call_iterator_t &>, std::__1::__list_iterator >, boost::signals2::slot &), boost::function &)> >, boost::signals2::mutex> >, void >, boost::signals2::detail::connection_body >, boost::signals2::slot &), boost::function &)> >, boost::signals2::mutex> > @ 0x0000700000414b40) const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >) const + 69 at optional_last_value.hpp:55
    frame update perf data host macros #26: 0x0000000100fc770f libbase.dylibvoid boost::signals2::detail::combiner_invoker::operator(this=0x0000700000414d00, combiner=0x00000001035b2560, first=slot_call_iterator_t &>, std::__1::__list_iterator >, boost::signals2::slot &), boost::function &)> >, boost::signals2::mutex> >, void *>, boost::signals2::detail::connection_body >, boost::signals2::slot &), boost::function &)> >, boost::signals2::mutex> > @ 0x0000700000414bc0, last=slot_call_iterator_t &>, std::__1::__list_iterator >, boost::signals2::slot &), boost::function &)> >, boost::signals2::mutex> >, void *>, boost::signals2::detail::connection_body >, boost::signals2::slot &), boost::function &)> >, boost::signals2::mutex> > @ 0x0000700000414be0), boost::signals2::detail::slot_call_iterator_t const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value&, boost::signals2::detail::slot_call_iterator_t const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t const&>, std::__1::__list_iterator >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body >, boost::signals2::slot const&), boost::function const&)> >, boost::signals2::mutex> >) const + 175 at result_type_wrapper.hpp:64 frame #27: 0x0000000100fc748e libbase.dylibboost::signals2::detail::signal_impl const&), boost::signals2::optional_last_value, int, std::__1::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator(this=0x00000001035b75b0, args=0x0000700000414fe0)(boost::intrusive_ptr const&) + 1006 at signal_template.hpp:242
    frame change user/service objDef order #28: 0x0000000100fbf6bf libbase.dylibboost::signals2::signal const&), boost::signals2::optional_last_value, int, std::__1::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator(this=0x00000001035b82c0, args=0x0000700000414fe0)(boost::intrusive_ptr const&) + 47 at signal_template.hpp:723 frame #29: 0x0000000100fbebc5 libbase.dylibicinga::Stream::SignalDataAvailable(this=0x00000001035b82a0) + 85 at stream.cpp:55
    frame Add "ldap" CheckCommand #30: 0x0000000101000440 libbase.dylibicinga::TlsStream::OnEvent(this=0x00000001035b82a0, revents=1) + 1600 at tlsstream.cpp:197 frame #31: 0x0000000101000892 libbase.dylibnon-virtual thunk to icinga::TlsStream::OnEvent(this=0x00000001035b82a0, revents=1) + 34 at tlsstream.cpp:127
    frame Add Agent System Requirements to Docs 10-icinga2-client.md #32: 0x0000000100fb0fac libbase.dylibicinga::SocketEvents::ThreadProc() + 2684 at socketevents.cpp:147 frame #33: 0x0000000100fb9ea6 libbase.dylibboost::detail::thread_data::run(this=0x00000001035cae80) + 22 at thread.hpp:116
    frame Update 22-appendix.md #34: 0x0000000100082715 libboost_thread-mt.dylibboost::(anonymous namespace)::thread_proxy(void*) + 53 frame #35: 0x00007fff834849b1 libsystem_pthread.dylib_pthread_body + 131
    frame Typo fix. #36: 0x00007fff8348492e libsystem_pthread.dylib_pthread_start + 168 frame #37: 0x00007fff83482385 libsystem_pthread.dylibthread_start + 13
    (lldb)

icinga-migration · 2015-11-02T16:32:43Z

Updated by gbeutner on 2015-11-02 16:32:43 +00:00

Also, FWIW, 2.3.x does not support HTTP requests.

icinga-migration · 2015-11-02T16:45:26Z

Updated by gbeutner on 2015-11-02 16:45:26 +00:00

Status changed from Feedback to Assigned
Assigned to changed from netphantm to gbeutner
Target Version set to 2.4.0

icinga-migration · 2015-11-02T16:50:03Z

Updated by Anonymous on 2015-11-02 16:50:03 +00:00

Status changed from Assigned to Resolved
Done % changed from 0 to 100

Applied in changeset 9c57589.

icinga-migration · 2015-11-06T16:33:00Z

Updated by gbeutner on 2015-11-06 16:33:00 +00:00

Subject changed from api call causes icinga2 to crash to Incorrect JSON-RPC message causes Icinga 2 to crash

icinga-migration · 2015-11-12T15:34:44Z

Updated by mfriedrich on 2015-11-12 15:34:44 +00:00

Backport? changed from TBD to No

icinga-migration closed this as completed Nov 2, 2015

icinga-migration added bug Something isn't working area/api REST API labels Jan 17, 2017

icinga-migration added this to the 2.4.0 milestone Jan 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dev.icinga.com #10495] Incorrect JSON-RPC message causes Icinga 2 to crash #3569

[dev.icinga.com #10495] Incorrect JSON-RPC message causes Icinga 2 to crash #3569

icinga-migration commented Oct 30, 2015

icinga-migration commented Oct 30, 2015

icinga-migration commented Oct 30, 2015

icinga-migration commented Oct 30, 2015

icinga-migration commented Nov 1, 2015

icinga-migration commented Nov 2, 2015

icinga-migration commented Nov 2, 2015

icinga-migration commented Nov 2, 2015

icinga-migration commented Nov 2, 2015

icinga-migration commented Nov 6, 2015

icinga-migration commented Nov 12, 2015

[dev.icinga.com #10495] Incorrect JSON-RPC message causes Icinga 2 to crash #3569

[dev.icinga.com #10495] Incorrect JSON-RPC message causes Icinga 2 to crash #3569

Comments

icinga-migration commented Oct 30, 2015

icinga-migration commented Oct 30, 2015

icinga-migration commented Oct 30, 2015

icinga-migration commented Oct 30, 2015

icinga-migration commented Nov 1, 2015

icinga-migration commented Nov 2, 2015

icinga-migration commented Nov 2, 2015

icinga-migration commented Nov 2, 2015

icinga-migration commented Nov 2, 2015

icinga-migration commented Nov 6, 2015

icinga-migration commented Nov 12, 2015