When openobserve as syslog server, tcp connection was closed repeatly almost every 60s #4067

kmephistoh · 2024-07-26T07:52:27Z

Which OpenObserve functionalities are the source of the bug?

ingestion

Is this a regression?

Yes

Description

When openobserve as syslog server, tcp connection was closed repeatly almost every 60s, client print these annoying logs. Even so, logs can still be collected.

Jul 26 15:34:16 DX-C9 rsyslogd: omfwd: remote server at 10.11.10.24:5514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2406.0 try https://www.rsyslog.com/e/2027 ]
Jul 26 15:34:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2406.0 try https://www.rsyslog.com/e/2007 ]
Jul 26 15:34:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2406.0 try https://www.rsyslog.com/e/2359 ]
Jul 26 15:34:16 DX-C9 rsyslogd: omfwd: remote server at 10.11.10.24:5514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2406.0 try https://www.rsyslog.com/e/2027 ]
Jul 26 15:34:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2406.0 try https://www.rsyslog.com/e/2007 ]
Jul 26 15:34:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2406.0 try https://www.rsyslog.com/e/2359 ]
Jul 26 15:34:16 DX-C9 rsyslogd: omfwd: remote server at 10.11.10.24:5514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2406.0 try https://www.rsyslog.com/e/2027 ]
Jul 26 15:34:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2406.0 try https://www.rsyslog.com/e/2007 ]
Jul 26 15:34:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2406.0 try https://www.rsyslog.com/e/2359 ]
Jul 26 15:35:16 DX-C9 rsyslogd: omfwd: remote server at 10.11.10.24:5514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2406.0 try https://www.rsyslog.com/e/2027 ]
Jul 26 15:35:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2406.0 try https://www.rsyslog.com/e/2007 ]
Jul 26 15:35:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2406.0 try https://www.rsyslog.com/e/2359 ]
Jul 26 15:36:16 DX-C9 rsyslogd: omfwd: remote server at 10.11.10.24:5514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2406.0 try https://www.rsyslog.com/e/2027 ]
Jul 26 15:36:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2406.0 try https://www.rsyslog.com/e/2007 ]
Jul 26 15:36:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2406.0 try https://www.rsyslog.com/e/2359 ]
Jul 26 15:36:16 DX-C9 rsyslogd: omfwd: remote server at 10.11.10.24:5514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2406.0 try https://www.rsyslog.com/e/2027 ]
Jul 26 15:36:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2406.0 try https://www.rsyslog.com/e/2007 ]
Jul 26 15:36:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2406.0 try https://www.rsyslog.com/e/2359 ]
Jul 26 15:37:16 DX-C9 rsyslogd: omfwd: remote server at 10.11.10.24:5514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2406.0 try https://www.rsyslog.com/e/2027 ]
Jul 26 15:37:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2406.0 try https://www.rsyslog.com/e/2007 ]
Jul 26 15:37:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2406.0 try https://www.rsyslog.com/e/2359 ]
Jul 26 15:37:16 DX-C9 rsyslogd: omfwd: remote server at 10.11.10.24:5514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2406.0 try https://www.rsyslog.com/e/2027 ]
Jul 26 15:37:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2406.0 try https://www.rsyslog.com/e/2007 ]
Jul 26 15:37:16 DX-C9 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2406.0 try https://www.rsyslog.com/e/2359 ]

When i select rsyslog as a syslog server on the same server 10.11.10.24:514, everything is ok(no annoying logs).

tcpdump show me, the connection was RST, so i think the reason may was the server side. no firewall or load balancer.

Please provide a link to a minimal reproduction of the bug

No response

Please provide the exception or error you saw

No response

Please provide the version you discovered this bug in (check about page for version information)

openobserve v0.10.8-rc5 
os: Ubuntu 22.04.4 LTS

Anything else?

No response

The text was updated successfully, but these errors were encountered:

gaby · 2024-07-26T12:44:07Z

I'm pretty sure this has to do with TCP keepalive, are you sending syslog during those 60seconds?

kmephistoh · 2024-07-29T02:24:00Z

I'm pretty sure this has to do with TCP keepalive, are you sending syslog during those 60seconds?

Sorry to reply too late, sometime it has log during this 60, The result is still the same。

As can be seen from the above test, the same log is printed each time.

doychi · 2024-08-21T22:03:12Z

I am getting similar issuse when logging to OpenObserver as a syslog server running under Unraid 6.12.11, except I'm getting more frequent errors:

Aug 22 07:52:47 unraid01 rsyslogd: omfwd: remote server at 192.168.3.172:1514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
Aug 22 07:52:47 unraid01 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Aug 22 07:52:47 unraid01 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Aug 22 07:52:49 unraid01 kernel: eth0: renamed from veth42b5b63
Aug 22 07:52:49 unraid01 rsyslogd: omfwd: remote server at 192.168.3.172:1514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
Aug 22 07:52:49 unraid01 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Aug 22 07:52:49 unraid01 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Aug 22 07:52:49 unraid01 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethb277e8f: link becomes ready
Aug 22 07:52:49 unraid01 kernel: docker0: port 2(vethb277e8f) entered blocking state
Aug 22 07:52:49 unraid01 kernel: docker0: port 2(vethb277e8f) entered forwarding state
Aug 22 07:53:12 unraid01 kernel: veth42b5b63: renamed from eth0
Aug 22 07:53:12 unraid01 rsyslogd: omfwd: remote server at 192.168.3.172:1514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
Aug 22 07:53:12 unraid01 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Aug 22 07:53:12 unraid01 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]
Aug 22 07:53:12 unraid01 kernel: docker0: port 2(vethb277e8f) entered disabled state
Aug 22 07:53:12 unraid01 kernel: docker0: port 2(vethb277e8f) entered disabled state
Aug 22 07:53:12 unraid01 kernel: device vethb277e8f left promiscuous mode
Aug 22 07:53:12 unraid01 kernel: docker0: port 2(vethb277e8f) entered disabled state
Aug 22 07:53:12 unraid01 rsyslogd: omfwd: remote server at 192.168.3.172:1514 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts down or aborts a connection. Rsyslog will re-open the connection if configured to do so (we saw a generic IO Error, which usually goes along with that behaviour). [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
Aug 22 07:53:12 unraid01 rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2102.0 try https://www.rsyslog.com/e/2007 ]
Aug 22 07:53:12 unraid01 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]

Signed-off-by: Yashodhan Joshi <[email protected]>

hengfeiyang · 2024-08-27T05:22:24Z

@kmephistoh @doychi do you mind we give you a dev version for help testing the fix?

doychi · 2024-08-27T05:27:52Z

Hi, I'd love to, but I'll have to work out how to get it installed under Unraid. Thanks, Justin Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Hengfei Yang ***@***.***> Sent: Tuesday, August 27, 2024 3:22:45 PM To: openobserve/openobserve ***@***.***> Cc: Doychi ***@***.***>; Mention ***@***.***> Subject: Re: [openobserve/openobserve] When openobserve as syslog server, tcp connection was closed repeatly almost every 60s (Issue #4067) @kmephistoh<https://github.com/kmephistoh> @doychi<https://github.com/doychi> do you mind we give you a dev version for help testing the fix? — Reply to this email directly, view it on GitHub<#4067 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACT6YFV3FDZBG5EPRSFNG6TZTQEKLAVCNFSM6AAAAABLQAI3BCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGU4TMMJUGE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

hengfeiyang · 2024-08-27T10:09:11Z

@doychi we only have docker image for dev version. maybe let's wait for some days, we will release a binary version then i will back to you.

kmephistoh · 2024-08-29T11:09:12Z

@kmephistoh @doychi do you mind we give you a dev version for help testing the fix?
ok, i'm waiting here.

hengfeiyang · 2024-08-29T11:29:18Z

@kmephistoh can you try this version:

public.ecr.aws/zinclabs/openobserve-dev:v0.11.0-rc3-e7e01c5

kmephistoh · 2024-08-30T02:03:01Z

public.ecr.aws/zinclabs/openobserve-dev:v0.11.0-rc3-e7e01c5

Can't download

hengfeiyang · 2024-08-30T10:10:47Z

@kmephistoh this is docker image, you can access it by browser, you can use

docker pull public.ecr.aws/zinclabs/openobserve-dev:v0.11.0-rc3-684dd39

or directly run it:

docker run -tdi --name o2-test -e ZO_ROOT_USER_EMAIL="[email protected]" -e ZO_ROOT_USER_PASSWORD="Complexpass#123" -e ZO_DATA_DIR="/data" -p 5080:5080 -p 5514:5514 public.ecr.aws/zinclabs/openobserve-dev:v0.11.0-rc3-684dd39

Please try this version, we added some debug logs.

doychi · 2024-08-30T22:46:00Z

I have installed the image and will keep an eye on it today and let you know how it goes.

doychi · 2024-09-01T02:32:31Z

The new image seems to have resolved the issue for me.

doychi · 2024-09-01T08:29:03Z

Can you let me know when this is likely to make it the latest stable release, so I can switch back to the stable line?

hengfeiyang · 2024-09-01T09:07:15Z

sure, will make release next week

kmephistoh · 2024-09-02T10:30:22Z

@kmephistoh this is docker image, you can access it by browser, you can use
docker pull public.ecr.aws/zinclabs/openobserve-dev:v0.11.0-rc3-684dd39
or directly run it:
docker run -tdi --name o2-test -e ZO_ROOT_USER_EMAIL="[email protected]" -e ZO_ROOT_USER_PASSWORD="Complexpass#123" -e ZO_DATA_DIR="/data" -p 5080:5080 -p 5514:5514 public.ecr.aws/zinclabs/openobserve-dev:v0.11.0-rc3-684dd39
Please try this version, we added some debug logs.

I try this version, the issue's problem was resolved. But other normal log are loss sometime.
Thank you for your help

hengfeiyang · 2024-09-02T10:43:21Z

We released v0.11.0, both of you can test it now.

kmephistoh · 2024-09-04T08:42:47Z

We released v0.11.0, both of you can test it now.

The problem was resolved, thanks for your help.

kmephistoh added the ☢️ Bug Something isn't working label Jul 26, 2024

hengfeiyang assigned hengfeiyang and YashodhanJoshi1 and unassigned YashodhanJoshi1 Aug 26, 2024

YashodhanJoshi1 pushed a commit that referenced this issue Aug 26, 2024

fix: keep tcp connections alive for syslog, fixes #4067

551fdcf

Signed-off-by: Yashodhan Joshi <[email protected]>

YashodhanJoshi1 mentioned this issue Aug 26, 2024

fix: keep tcp connections alive for syslog #4335

Merged

hengfeiyang closed this as completed in #4335 Aug 27, 2024

hengfeiyang assigned neha00290 Aug 27, 2024

hengfeiyang added the testing In Testing label Aug 27, 2024

hengfeiyang reopened this Aug 27, 2024

hengfeiyang mentioned this issue Aug 30, 2024

fix: debug syslog ingestion #4402

Merged

hengfeiyang unassigned hengfeiyang and YashodhanJoshi1 Sep 3, 2024

hengfeiyang closed this as completed Sep 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When openobserve as syslog server, tcp connection was closed repeatly almost every 60s #4067

When openobserve as syslog server, tcp connection was closed repeatly almost every 60s #4067

kmephistoh commented Jul 26, 2024

gaby commented Jul 26, 2024

kmephistoh commented Jul 29, 2024

doychi commented Aug 21, 2024

hengfeiyang commented Aug 27, 2024

doychi commented Aug 27, 2024 via email

hengfeiyang commented Aug 27, 2024

kmephistoh commented Aug 29, 2024

hengfeiyang commented Aug 29, 2024

kmephistoh commented Aug 30, 2024

hengfeiyang commented Aug 30, 2024

doychi commented Aug 30, 2024

doychi commented Sep 1, 2024

doychi commented Sep 1, 2024

hengfeiyang commented Sep 1, 2024

kmephistoh commented Sep 2, 2024 •

edited

Loading

hengfeiyang commented Sep 2, 2024

kmephistoh commented Sep 4, 2024 •

edited

Loading

When openobserve as syslog server, tcp connection was closed repeatly almost every 60s #4067

When openobserve as syslog server, tcp connection was closed repeatly almost every 60s #4067

Comments

kmephistoh commented Jul 26, 2024

Which OpenObserve functionalities are the source of the bug?

Is this a regression?

Description

Please provide a link to a minimal reproduction of the bug

Please provide the exception or error you saw

Please provide the version you discovered this bug in (check about page for version information)

Anything else?

gaby commented Jul 26, 2024

kmephistoh commented Jul 29, 2024

doychi commented Aug 21, 2024

hengfeiyang commented Aug 27, 2024

doychi commented Aug 27, 2024 via email

hengfeiyang commented Aug 27, 2024

kmephistoh commented Aug 29, 2024

hengfeiyang commented Aug 29, 2024

kmephistoh commented Aug 30, 2024

hengfeiyang commented Aug 30, 2024

doychi commented Aug 30, 2024

doychi commented Sep 1, 2024

doychi commented Sep 1, 2024

hengfeiyang commented Sep 1, 2024

kmephistoh commented Sep 2, 2024 • edited Loading

hengfeiyang commented Sep 2, 2024

kmephistoh commented Sep 4, 2024 • edited Loading

kmephistoh commented Sep 2, 2024 •

edited

Loading

kmephistoh commented Sep 4, 2024 •

edited

Loading