failed to purge snap db file #18124

kelen121 · 2024-06-04T08:51:25Z

kelen121
Jun 4, 2024

Bug report criteria

This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
You have read the etcd bug reporting guidelines.
Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

etcd exit, log:
{"level":"fatal","ts":"2024-06-01T22:11:59.898283 0800","caller":"etcdserver/server.go:887","msg":"failed to purge snap db file","error":"open /data/trustworkspace/data/etcd/member/snap: permission denied","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).purgeFile\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:887\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2761"}

What did you expect to happen?

why exit?

How can we reproduce it (as minimally and precisely as possible)?

The etcd process exits irregularly.

Anything else we need to know?

No response

Etcd version (please run commands below)

etcd Version: 3.5.12
Git SHA: e7b3bb6
Go Version: go1.20.13
Go OS/Arch: linux/amd64

Etcd configuration (command line flags or environment variables)

name: 'default'

Path to the data directory.

data-dir: /data/trustworkspace/data/etcd

Path to the dedicated wal directory.

wal-dir: /data/trustworkspace/data/etcd-wal

#snapshot: false

Number of committed transactions to trigger a snapshot to disk.

snapshot-count: 1000000000000

Time (in milliseconds) of a heartbeat interval.

heartbeat-interval: 100

Time (in milliseconds) for an election to timeout.

election-timeout: 1000

Raise alarms when backend size exceeds the given quota. 0 means use the

default quota.

quota-backend-bytes: 0

List of comma separated URLs to listen on for client traffic.

listen-client-urls: http://10.21.20.209:2379

Maximum number of snapshot files to retain (0 is unlimited).

max-snapshots: 5

Maximum number of wal files to retain (0 is unlimited).

max-wals: 5

List of this member's client URLs to advertise to the public.

The URLs needed to be a comma-separated list.

advertise-client-urls: http://******:2379

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

No response

Relevant log output

No response

kelen121 · 2024-06-04T08:53:04Z

kelen121
Jun 4, 2024
Author

{"level":"warn","ts":"2024-06-01T21:00:29.370356 0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"KMrdlShHiPnPkfzs.3120"}
{"level":"info","ts":"2024-06-01T21:05:30.474473 0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"fSlTYFvrhGsYotQp.3121"}
{"level":"warn","ts":"2024-06-01T21:10:58.003982 0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"fSlTYFvrhGsYotQp.3121"}
{"level":"info","ts":"2024-06-01T21:15:58.473863 0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"WjwHoyppYGeOgXWn.3122"}
{"level":"warn","ts":"2024-06-01T21:21:26.639016 0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"WjwHoyppYGeOgXWn.3122"}
{"level":"info","ts":"2024-06-01T21:26:27.473623 0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"nvRPEgXByBrJKfun.3123"}
{"level":"warn","ts":"2024-06-01T21:31:55.271978 0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"nvRPEgXByBrJKfun.3123"}
{"level":"info","ts":"2024-06-01T21:36:55.474155 0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"rlqIsSqsEwSsmMas.3124"}
{"level":"warn","ts":"2024-06-01T21:42:23.907282 0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"rlqIsSqsEwSsmMas.3124"}
{"level":"info","ts":"2024-06-01T21:47:24.474196 0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"QphKDbYtErENSquF.3125"}
{"level":"warn","ts":"2024-06-01T21:52:52.54077 0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"QphKDbYtErENSquF.3125"}
{"level":"info","ts":"2024-06-01T21:57:53.474383 0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"gQGiCpHxEkOeaTvf.3126"}
{"level":"warn","ts":"2024-06-01T22:03:21.176855 0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"gQGiCpHxEkOeaTvf.3126"}
{"level":"info","ts":"2024-06-01T22:08:21.473633 0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"IMSxAQofTGOcJShk.3127"}
{"level":"fatal","ts":"2024-06-01T22:11:59.898283 0800","caller":"etcdserver/server.go:887","msg":"failed to purge snap db file","error":"open /data/trustworkspace/data/etcd/member/snap: permission denied","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).purgeFile\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:887\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2761"}

3 replies

jmhbnz Jun 4, 2024
Maintainer

Hi @kelen121 - As shown in the error logs above, the etcd server process is exiting because it cannot write the snapshot file. Please check and confirm which user the process is running under, and that the user can write to the file.

If the issue persists please provide clear step by step reproduce instructions so we can attempt to reproduce the same scenario at our end. Thanks.

kelen121 Jun 6, 2024
Author

I am running as the trustworkspace user and have already granted the user 777 permissions, as follows:
[root@tws-dev logs]# ll /data/trustworkspace/data/etcd/member/snap
总用量 16408
-rwxrwxrwx. 1 trustworkspace trustworkspace 17166336 5月 31 17:54 db

jmhbnz Jun 9, 2024
Maintainer

Thanks for the update @kelen121. Is this issue limited to a single machine or can you recreate it elsewhere?

Please provide step by step process to recreate the issue so we can attempt it on our side.

kelen121 · 2024-06-04T08:54:17Z

kelen121
Jun 4, 2024
Author

[root@tws-dev logs]# ll /data/trustworkspace/data/etcd/member/snap
总用量 16408
-rwxrwxrwx. 1 trustworkspace trustworkspace 17166336 5月 31 17:54 db

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

failed to purge snap db file #18124

{{title}}

Path to the data directory.

Path to the dedicated wal directory.

Number of committed transactions to trigger a snapshot to disk.

Time (in milliseconds) of a heartbeat interval.

Time (in milliseconds) for an election to timeout.

Raise alarms when backend size exceeds the given quota. 0 means use the

default quota.

List of comma separated URLs to listen on for client traffic.

Maximum number of snapshot files to retain (0 is unlimited).

Maximum number of wal files to retain (0 is unlimited).

List of this member's client URLs to advertise to the public.

The URLs needed to be a comma-separated list.

Replies: 2 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

failed to purge snap db file #18124

kelen121 Jun 4, 2024

Bug report criteria

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Etcd version (please run commands below)

Etcd configuration (command line flags or environment variables)

Path to the data directory.

Path to the dedicated wal directory.

Number of committed transactions to trigger a snapshot to disk.

Time (in milliseconds) of a heartbeat interval.

Time (in milliseconds) for an election to timeout.

Raise alarms when backend size exceeds the given quota. 0 means use the

default quota.

List of comma separated URLs to listen on for client traffic.

Maximum number of snapshot files to retain (0 is unlimited).

Maximum number of wal files to retain (0 is unlimited).

List of this member's client URLs to advertise to the public.

The URLs needed to be a comma-separated list.

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Relevant log output

Replies: 2 comments · 3 replies

kelen121 Jun 4, 2024 Author

jmhbnz Jun 4, 2024 Maintainer

kelen121 Jun 6, 2024 Author

jmhbnz Jun 9, 2024 Maintainer

kelen121 Jun 4, 2024 Author

kelen121
Jun 4, 2024

Replies: 2 comments 3 replies

kelen121
Jun 4, 2024
Author

jmhbnz Jun 4, 2024
Maintainer

kelen121 Jun 6, 2024
Author

jmhbnz Jun 9, 2024
Maintainer

kelen121
Jun 4, 2024
Author