Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting Frequent dismount issue of OCI object Storage #2457

Open
kparmar-winfo opened this issue May 23, 2024 · 9 comments
Open

Getting Frequent dismount issue of OCI object Storage #2457

kparmar-winfo opened this issue May 23, 2024 · 9 comments
Labels
need info Waiting for reporter feedback

Comments

@kparmar-winfo
Copy link

kparmar-winfo commented May 23, 2024 •
edited
Loading

Additional Information

Version of s3fs being used (s3fs --version)

V1.93 (commit:unknown) with Open SSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

Name : fuse
Version : 2.9.4
Release : 1.0.9.el7

Kernel information (uname -r)

5.4.17-2136.321.4.el7uek.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Oracle Linux Server"
VERSION="7.9"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Oracle Linux Server 7.9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:9:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.9
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.9

How to run s3fs, if applicable

[] command line
[] /etc/fstab

# MOUNT OBJECT STORE BUCKET AS DISK
s3fs#obj-demo-standard /objstore/demo fuse _netdev,allow_other,use_path_request_style,nomultipart,umask=0000,uid=0,gid=54322,endpoint=uk-london-1,url=https://<NAMESPACE>.compat.objectstorage.uk-london-1.oraclecloud.com/ 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Details about issue

Currently we are using Oracle Linux 7 system in which we have mounted an OCI object storage using s3fs-fuse utility. But since few months we are facing an issue as mountpoint dismounted frequetly with segement fault error.

Please check the below log, generated during dismount of the object storage in /var/log/messages file.

May 19 10:36:03 test01 kernel: s3fs[19840]: segfault at 0 ip 000055a7e80b01ed sp 00007efd2e7fb8c0 error 4 in s3fs[55a7e80a0000 9d000]
May 19 10:36:03 test01 kernel: Code: 00 00 00 48 89 44 24 38 31 c0 49 8d 44 24 18 48 89 44 24 20 31 c0 e8 b2 35 01 00 48 8d 3d 73 ce 06 00 b9 02 00 00 00 48 89 ee a6 74 23 48 8d 3d cc 4e 07 00 b9 02 00 00 00 48 89 ee f3 a6 74
May 19 10:36:03 test01 abrt-hook-ccpp: Process 19698 (s3fs) of user 0 killed by SIGSEGV - dumping core
May 19 10:36:07 test01 abrtd: Size of '/var/spool/abrt' >= 5000 MB (MaxCrashReportsSize), deleting old directory 'ccpp-2024-05-16-10:31:44-18344'

We have also tried to update the s3fs-fuse to it's latest version, though we falll into the same place.

Thank you,
Karan Parmar.
@kparmar-winfo
Copy link
Author

We are frequently getting an issue. Can anyone respond to this issue please.

@gaul
Copy link
Member

gaul commented Jun 14, 2024

Please attach gdb to the s3fs process. When the segfault occurs, use thread apply all bt to get the backtrace.

@gaul gaul added the need info Waiting for reporter feedback label Jun 14, 2024
@gaul
Copy link
Member

gaul commented Jun 14, 2024

If you can tolerate the slower performance, valgrind will give an even better diagnostic.

@kparmar-winfo
Copy link
Author

kparmar-winfo commented Jun 17, 2024 •
edited
Loading

s3fs_gdb_backtrace.txt

Hi Gaul,

Have attached s3fs backtrace. Can you plerase check the output once.
Moreover I will try to diagnose process using valgrind and attach that as well later.

Also, I have checked /var/log/messages file in which I found below message. Not sure whether it is showing any error or not.

Jun 17 10:59:55 vm01 s3fs[27838]: Loaded mime information from /etc/mime.types
Jun 17 10:59:55 vm01 s3fs[27851]: init v1.94(commit:unknown) with OpenSSL, credential-library(built-in)
Jun 17 11:00:01 vm01 syst

@kparmar-winfo
Copy link
Author

valgrind-out1.txt

Hi Gaul,

We have ran the valgrind for the process of s3fs and attaching the same log here. Can you please check the log once.

@gaul
Copy link
Member

gaul commented Jun 17, 2024

gdb doesn't show a segfault -- are you sure you attached to the process that is failing? The valgrind output is more interesting:

==27851== Invalid read of size 1
==27851==    at 0x11A21C: ??? (in /usr/bin/s3fs)
==27851==    by 0x123C45: ??? (in /usr/bin/s3fs)
==27851==    by 0x4E4D8A6: ??? (in /usr/lib64/libfuse.so.2.9.4)
...
==27851==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

But it appears that s3fs lacks debug symbols so I don't know where the failure is occuring. Which distribution do you have? Can you install debug symbols for s3fs separately? Otherwise you will need to compile s3fs locally to make sure the -g compiler flag is specified so we can make progress.

@kparmar-winfo
Copy link
Author

An error message is reported in dmesg:

[root@vm01~]# dmesg|tail
[14234014.770576] s3fs[30277]: segfault at 0 ip 000058636b24521c sp 00007f8884ff8890 error 4 in s3fs[58636b233000 b8000]
[14234014.770863] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74
[14252238.490976] s3fs[13801]: segfault at 0 ip 000055c1deeb921c sp 00007f4edffbe890 error 4 in s3fs[55c1deea7000 b8000]
[14252238.490983] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74
[14253956.891204] s3fs[22090]: segfault at 0 ip 000055d4af4c821c sp 00007fc43124e890 error 4 in s3fs[55d4af4b6000 b8000]
[14253956.891220] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74
[14266639.509252] s3fs[4421]: segfault at 0 ip 00005596be0fb21c sp 00007f41fa7fb890 error 4 in s3fs[5596be0e9500 b8000]
[14266639.509270] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74
[14337729.456076] s3fs[2395]: segfault at 0 ip 0000561d8d0f521c sp 00007fcd977fd890 error 4 in s3fs[561d8d0e3000 b8000]
[14337729.456083] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74

In addition, I have added -o dbglevel=debug option while mounting s3fs bucket using valgrind. I will send it to you once error occur.

Also, not sure why am I getting commit as unknow:

[root@vm01~]# s3fs --version
Amazon Simple Storage Service File System V1.94 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <[email protected]>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[root@vm01~]#

@ggtakec
Copy link
Member

ggtakec commented Jun 23, 2024

@kparmar-winfo

In addition, I have added -o dbglevel=debug option while mounting s3fs bucket using valgrind. I will send it to you once error occur.

Thanks. I hope the log you get gives some hints.

Also, not sure why am I getting commit as unknow:

I think the reason why unknown is displayed is because the distributed package does not read the git commit hash.
The s3fs distribution package displays this.
(If you built it yourself, you can display the commit hash value if you have the git information.)
This display is misleading, so I will submit a PR to improve it soon.

@ggtakec ggtakec mentioned this issue Jun 23, 2024
Merged
@kparmar-winfo
Copy link
Author

Hi @ggtakec,

Did we got any progress on the above issue.

Thanks in advance.

@gaul gaul changed the title Gerttting Frequent dismount issue of OCI object Storage Getting Frequent dismount issue of OCI object Storage Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
need info Waiting for reporter feedback
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gaul @ggtakec @kparmar-winfo