Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Endless election cycle #21380

Open
nurmukhametdaniyar opened this issue Jun 28, 2024 · 0 comments
Open

Endless election cycle #21380

nurmukhametdaniyar opened this issue Jun 28, 2024 · 0 comments

Comments

@nurmukhametdaniyar
Copy link

Overview of the Issue

Starting a cluster with 3 nodes results in endless election cycle

Consul info for both Client and Server

Server info

Consul.json file

{
  "server": true,
  "bootstrap_expect": 3,
  "data_dir": "/var/consul",
  "bind_addr": "10.35.0.8",
  "retry_join": [ "10.30.0.13", "10.35.0.8", "10.36.0.12"],
  "ui": true
}
agent:
	check_monitors = 0
	check_ttls = 0
	checks = 0
	services = 0
build:
	prerelease = 
	revision = bf0166d8
	version = 1.19.0
	version_metadata = 
consul:
	acl = disabled
	bootstrap = false
	known_datacenters = 1
	leader = false
	leader_addr = 
	server = true
raft:
	applied_index = 0
	commit_index = 0
	fsm_pending = 0
	last_contact = never
	last_log_index = 181
	last_log_term = 210
	last_snapshot_index = 0
	last_snapshot_term = 0
	latest_configuration = [{Suffrage:Voter ID:9b79f23f-c83e-61ad-f61c-bd45ad20b15f Address:10.36.0.12:8300} {Suffrage:Voter ID:a0965b21-4900-5154-8a71-b5b06c4725e8 Address:10.30.0.13:8300}]
	latest_configuration_index = 0
	num_peers = 1
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Candidate
	term = 1439
runtime:
	arch = amd64
	cpu_count = 4
	goroutines = 136
	max_procs = 4
	os = linux
	version = go1.22.4
serf_lan:
	coordinate_resets = 0
	encrypted = false
	event_queue = 0
	event_time = 5
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 59
	members = 3
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = false
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 38
	members = 3
	query_queue = 0
	query_time = 1

Server agent HCL config

Operating system and Environment details

Ubuntu 20.04

Log Fragments

Jun 28 14:31:23 resticloud consul[2768931]: 2024-06-28T14:31:23.835 0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Jun 28 14:31:25 resticloud consul[2768931]: 2024-06-28T14:31:25.640 0500 [WARN]  agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:25 resticloud consul[2768931]: 2024-06-28T14:31:25.640 0500 [INFO]  agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1462
Jun 28 14:31:34 resticloud consul[2768931]: 2024-06-28T14:31:34.718 0500 [WARN]  agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:34 resticloud consul[2768931]: 2024-06-28T14:31:34.718 0500 [INFO]  agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1463
Jun 28 14:31:40 resticloud consul[2768931]: 2024-06-28T14:31:40.469 0500 [ERROR] agent: Coordinate update error: error="No cluster leader"
Jun 28 14:31:42 resticloud consul[2768931]: 2024-06-28T14:31:42.660 0500 [WARN]  agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:42 resticloud consul[2768931]: 2024-06-28T14:31:42.660 0500 [INFO]  agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1464
Jun 28 14:31:48 resticloud consul[2768931]: 2024-06-28T14:31:48.113 0500 [WARN]  agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:48 resticloud consul[2768931]: 2024-06-28T14:31:48.113 0500 [INFO]  agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1465
Jun 28 14:31:55 resticloud consul[2768931]: 2024-06-28T14:31:55.647 0500 [WARN]  agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:55 resticloud consul[2768931]: 2024-06-28T14:31:55.647 0500 [INFO]  agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1466
Jun 28 14:31:57 resticloud consul[2768931]: 2024-06-28T14:31:57.767 0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Jun 28 14:32:03 resticloud consul[2768931]: 2024-06-28T14:32:03.838 0500 [WARN]  agent.server.raft: Election timeout reached, restarting election
Jun 28 14:32:03 resticloud consul[2768931]: 2024-06-28T14:32:03.838 0500 [INFO]  agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1467
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nurmukhametdaniyar